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NSF DIRECTOR'S STATEMENT ON THE 
REPORT OF THE ADVISORY COMMITTEE ON 
MERIT REVIEW 


In April 1985, acting on @ recommendation of the National Science 
Board Committee on Excellence, I appointed an Advisory Committee 
on Merit Review, chaired by Dr. Norman Hackerman. I asked that 
it examine competitive review procedures at the National Science 
Foundation and at other Federal agencies and recommend changes 
and improvements, if appropriate. 


RATIONALE: 


The rapidly changing research environment and increased pressures 
on the review system suggested the need for a special committee 
of distinguished outside evaluators to examine Federal agency 
peer review procedures with an in-depth review of the system, 
focusing on the National Science Foundation. 


THE MERIT REVIEW SYSTEM 


Since its establishment, the National Science Foundation has 
relied on the judgment of qualified, experienced scientists and 
engineers to select the most promising high quality research for 
support. This system has been subjected to periodic examination 
by internal task forces and outside panels, including 
Congressional Committees. It has been found to be remarkably 
effective and flexible, adapting to the changing needs of science 
and engineering research, and incorporating improvements with 
respect to openness, accountability, equity, and impact on the 
research environment. 


As the core of the decision-making system that provides support 
to researchers at universities across the range of scientific 
disciplines, the competitive merit review system has served as a 
barometer -- and even a lightning rod -- of change in the 
research system. The expansion of research opportunities and 
steady growth in the number of high caliber research proposals 
have sharply increased the competitive pressure on the resources 
available to support research and the research infrastructure. 
This in turn, has increased pressure on the merit review system. 


NSF iecognizes that shortfalls in support for academic science 
and engineering research facilities have led to pressures on the 
competitive merit review system. This situation has in some 
instances led to activities that bypass the merit review system. 
The Advisory Committee's report emphasizes that the existence of 
this situation does not imply unsoundness in the review system 
itself. Rather, the report strongly reaffirms the fundamental 
health of the present system. The Foundation does not agree with 
suggestions in the report that additional review mechanisms are 
needed or would be efficacious in addressing the problem of 
bypassing merit review. 


NSF TASK FORCE 


A task force chaired by NSF's Senior Science Advisor, Dr. Mary 
Clutter, was charged with the responsibility to undertake a 
careful review of the Committee's conclusions and recommendations 
and to develop an action plan for the Foundation. The action 
plan that follows is besed on that review. 


ACTION PLAN 


The task force has aggregated the Advisory Committee's recom- 
mendations into an ection plan consisting of five categories. 
I have reviewed the plan and intend to implement it immediately. 


SELECTION CRITERIA: NSF's primary criteria for selection of 
research projects have always been the quality of the proposed 
research and the competence of the investigators. Once 
excellence has been established secondary criteria are applied. 
Specifically, attention is given to the effect of the project on 
the research infrastructure, and to contributions to related 
goals of equity and distribution of resources among institutions 
and geographic areas. The changing scale and organization of 
science have increased the importance of these factors and of 
multiple elements in the evaluation of a proposal. This is 
especially true for center-based activities, research groups and 
shared facilities. 


1. The Advisory Committee recognized the evolution of the peer 
review process and recommended adopting the term Merit Review to 
describe the selection process. (II-1) 


Action: Merit Review is a more accurate descripticn of the 
review process employed at NSF. The recommendation of the 
Committee to adopt its use will be implemented, and top 
level management will insure compliance in application of 
the extended criteria to appropriate programs and proposals. 


2. Mulitidisciplinary proposals and other large projects require 
Gifferent evaluation procedures from those normally used for 
Single investigator proposals. (II-5) 


Action: NSF has developed review procedures such as the use 
of multi-stage review panels to handle evaluation of large 
projects, for example, Engineering Research Centers. These 
procedures need to be codified and will be applied to other 
large project proposals, as appropriate. 


3. The Committee also addressed the evaluation of standard 
proposals that are interdisciplinary or fall between internal 
Organizational units. (II-6) 
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Action: Interdisciplinary panels, as currently used in 
programs such as the Chemistry of Life Processes, will be 
considered for other new and rapidly evolving interdisci- 
plinary research areas. 


Action: Unsolicited interdisciplinary research proposals 
from individual investigators that do not fall readily into 
established program areas occasionally present review 
problems. An ad hoc task group is currently formulating 
recommendations for the processing and review of such 
proposals. 


4. The fairness of the review process is of prime importance. 

This can only be assured if reviewers and review panels reflect 
the broad community of excellent scientists and engineers from 

academia, government, and industry. (II-7) 


Action: The Foundation has in place specific guidelines for 
the inclusion of women and minority scientists or engineers 
and individuals from predominantly undergraduate 
institutions in the review process including participation 
on review panels and advisory committees. Management will 
follow up to insure that practice at the program level is in 
full compliance with these guidelines. 


Creativity and Risk: The support of excellence in research must 
include receptivity to creative, high-risk research. NSF has 
always encouraged its program officers to support such proposals. 
(II-9) 


Action: NSF will encourage program managers to make greater 
use of discretionary authority to identify and support 
creative, high-risk proposals. Special allocation of funds 
will be considered pending evaluation of a current 
experimental program. 


Action: Oversight revie. committees and disciplinary 
advisory committees will be instructed to identify and 
examine creative high-risk proposals, both funded and 
declined, and to report the extent to which they are given 
appropriate consideration. 


Data Bases: Effective monitoring of the review process by 
Management rests on the adequacy of the information available on 
it. NSF data bases have been constructed to include information 
on proposal submission and awards and on reviewers and advisory 
committee members by sex, age, institutional affiliation, 
minority group and disabled/handicapped status. (II-2) 


Action: Procedures will be established to facilitate 
periodic management review of the accuracy and completeness 
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of existing data bases and specifically to determine if the 
existing data bases need to be improved or expanded. 


Action: At the conclusion of each fiscal year, a report 
will be prepared summarizing the attributes of the proposers 
and reviewers, providing insight into the changes in the 
reviewer and proposer pools over time. 


Responsiveness: The equity and accountability of the review 
process relies in good measure on its transparency and 
timeliness, and its responsiveness to the needs of the research 
community. (I11I-3,4) 


1. Feedback to Reviewers 
Action: Responsible reviewers willing to provide their 
expertise on a repeated basis are the core of the review 
process. NSF will provide feedback to them by sending them 
a list of all awards made in disciplinary areas they 
reviewed at the conclusion of each fiscal year. 


2. Reviewer Responsibility 
Action: Reviewers bear a responsibility to render a 
balanced judgment, free of bias and vituperation. 
Incomplete review and intemperate or personal remarks are 
unacceptable and will not be used in the selection process. 
Futhermore, principal investigators will be notified that 
such reviews were not used and given an explanation. 
Finally, extensive education of the reviewer community to 
NSF's requirements will be undertaken. 


3. Timeliness 
Action: NSF will accentuate its efforts to reduce proposal 
processing time within flexible parameters that acknowledge 
the special demands of complex proposals. For the majority 
of proposals, the current goal of six months will remain in 
effect. 


Infrastructure: The increasing need to improve the 
infrastructure of science and engineering has concomitantly 
increased competitive pressure for resources available for 
support. 


1. Facilities - Pressure is particularly acute in areas where 
federal investment has decreased over the past 20 years, namely 
in the support of facilities construction and renovation. The 
appropriate federal role with regard to facilities support 
remains to be determined and the Foundation will continue to 
cooperate with this effort. (II-8) 


Action: NSF is mandated to provide on a biennial basis, the 
database on facilities to the Congress. The first report 
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has now been released, and should prove helpful in defining 
the appropriate federal role in support of facilities. 


In addition, NSF has in place a mechanism to respond to 
facilities needs within the context of research proposals. On 
September 27, 1985 an Important Notice was released to the entire 
university and college research community clarifying the 
Foundation's policy on construction and renovation of research 
and education facilities. 


2. Capacity Building - The Advisory Committee recommended the 
adoption of long term preferential programs to improve the 
science and engineering base throughout the U.S. and was 
concerned about equitable review. NSF has in place a number of 
capacity-building programs, for example, Experimental Program to 
Stimulate Competitive Research (EPSCOR), Minority Research 
Initiation (MRI), College Science Instrumentation Program (CSIP). 
Special review procedures insure equitable treatment. (II-7) 


Action: NSF will review all capacity building programs and 
establish evaluation criteria to monitor effectiveness and 
progress towards the ultimate goal of mainstreaming such 
activities. 


CONCLUSION 


The Advisory Committee on Merit Review is to be commended on the 
successful completion of its challenging assignment. The 
recommendations of the Committee that frame the agenda for future 
action are an important contribution to our continued efforts to 
enhance the effectiveness of NSF's review procedures. As the 
Advisory Committee's report points out, the dynamism and 
flexibility of the competitive merit review system have been 
critical elements of its success. It must remain receptive to 
change and improvement in the future if it is to continue to 
serve the needs of the Nation. 


aor“ 
Erich Bloch 
December 17, 1986 


September 25, 1986 


Mr. Erich Bloch, Director 
National Science Foundation 
Washington, D.C. 20550 


Dear Erich: 


I am pleased to transmit to you the final report of your 
Advisory Committee on Merit Review. 


Our mandate was broad, and the subject complex. Accordingly, 
we set our focus at three levels. 


First, and most important, the report defines the concept of 
merit review, describes the process as practiced in different 
Federal agencies and programe, and offers an historical 
explanation concerning the evolution from “peer” review to the 
more comprehensive merit review process. 


Second, it examines review processes in several NSF program 
areas in which the criteria for project selection include both 
technical excellence as well as non-technical concerns such as 
building research infrastructure or the likelihood of useful 
applications. Model review processes for different kinds of 
objectives are identified, and recommendations are made. 


Finally. it examines in detail several issues in carrying 
out the two principal forms of project review at NSF -- the ad 
hoc mail review, and the panel review processes. The role of the 
program manater and problems of accountability are also 
discussec. Several improvements are recommended. 


The report wes adopted by the Committee without dissent. The 
Committee thinks the report is informative, and believes it will 
be useful tw you, the science and engineering communities, and 
ecience policy makers. 


peo 


oyman Hackerman, Chairman 
isory Committee on 
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PREFACE 


Committee Background and Charge 


In 1985 the National Science Board issued a report by its 
Committee on Excellence in Science and Engineering on the 
prcblem of direct Congressional funding of academic research 
facilities, by-passing the merit review system. One of its 
three recommendations was that the NSF Director should 
appoint a committee to examine project selection procedures 
within the Naticral Science Foundation, survey other agencies' 
approaches, and .ecommend modifications of procedures if 
appropriate. 


In response the NSF director appointed an 11 member Advisory 
Committee on Merit Review in April 1985. The charge to the 
Committee read as follows: 


“Purpose and Functions 


A. Purpose: The Advisory Committee on Merit Review will 
evaluate merit review as practised by NSF and other agencies 
and provide its advice and recommendations concerning alter- 
native systems of merit review and selection of projects for 
grants. 


B. Functions: The principal functions of this Committee are <o: 


1. Document present methods of merit review at NSF and other 
appropriate agencies. This includes vLoth peer review, 
per se, as well as selection criteria used in decision- 
making. 

2. Evaluate differences between the implementation of merit 
review among NSF directorates. 

3. Analyze how merit review applies to diverse activities, 
such as general research grants, instrumentation awards, 
facilities, and special programs (e.g., international, 
women, and minority programs). 

4. Assess inequities that may result from merit review, such 
as underrepresentation of certain institutions or consti- 
tuencies. 

5S Identify the strengths and shortcomings of present pro- 
cedures. 

6. Make recommendations for overcoming limitations of the 
present methods of merit review. 

7. Prepare reports of its activities as appropriate." 


The Cemmittee met five times -- in July, October and November 
1985 and February and July i986. It heard presentations from 
all the NSF directorates, from representatives of major 

Federal research supporting agencies (DoD, NIH, DoE and 

USDA), and from staff of the House Subcommittee on Science, 
Research and Technology. The Committee also received and 
considereu correspondence from a variety of interested parties. 
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I. EXECUTIVE SUMMARY AND CONCLUSIONS 


1. Overall Assessment (See III and VI A, below) 


Technical (peer) review is a cornerstone of Federal 
agency arrangements for research project selection. The 
Committee found that by and large the system is functioning 
well. No evidence was found that standards of technical 
excellence were being compromised in the review processes 
of the various Federal agencies, including the Nationa] 
Science Foundation, although improvements are possible and 
desirable. 


For an increasing proportion of Federally sponsored 
research, however, technical excellence is a necessary but 
not fully sufficient criterion for research funding. To 
reach goals such as increasing the practical relevance of 
research results, or improving the nation's infrastructure 
for science and engineering, additional criteria are needed. 
The wide range of mechanisms evolved by Federal agencies for 
utilizing such additional criteria is impressive. The Con- 
mittee has adopted the term “merit review" to refer to selec- 
tion processes which include technical as well as these ad- 
ditional considerations. The Committee's discussion focusses 
both on narrower issues for improvement of technical (peer) 
review as well as on broader merit considerations. 


Improving the Process and Quality of Review (See VI B below) 


a) Reviewer Pool (See VI Bl, below) 


NSF's reviewer pool is too large and diverse to warrant 
charges of elitism or in-group exclusiveness. Nevertheless, 
a number of important issues need to be addressed. How 
reviewers get into the pool, how long they stay, how often 
they are requested to review, how often they decline to 
review, are all important questions that can only be 
addressed anecdotally at present. The Committee recommends 
a serious effort to develop data covering these issues. 


b) Promptness (See VI B2, below) 


While increasing program manager discretion will speed 
some proposal decisions, measures to improve merit consid- 
eration of proposals may increase proposal processing time 
fur others. Continued efforts should be made to reduce 
further administrative processing time for proposals. 

The “ommittee encourages NSF to strive for its current 
goal which is six months maximum processing time. 


c) Quality of Mail Reviews (See VI B3, below) 


The review process can be no stronger than the willingness 
of the community to participate in it with vigor and 
integrity. Anecdotes suggest that poor quality reviews -- 
sometimes perfunctory, sometimes intemperate -- do occur. 
The Committee recommends that the language of NSF's in- 
structions to reviewers be revised to include requests for 
full coverage and moderate language. The extent of ir- 
responsible reviewer behavior should also be examined, and 
program managers should be given every support possible to 
help them select the best reviewers. 


The Committee also recommends that means be explored to 
provide reviewers with some kind of feedback on their 
contributions, such as sending them the annual listing 
of program awards. 


dad) Panel versus Ad Hoc Mail Review (See VI B4, below) 


Non-scientists, and even scientists and engineers themselves, 


find the paraliel existence of two basic forms of project 
review puzzling. The Committee recognizes that each basic 
type of review is rooted in the traditions of different 
research communities, and that each has certain strengths 
and vulnerabilities. No strong preference was expressed 
for either model. Perhaps, like corn, well thought out 
and nurtured hybrid varieties promise the most vigorous 
yielcs for the future. 


e) Conflict of Interests (See VI B5, below) 


Some conflict of interests is inherent in any process of 
review by peers. The Committee believes, however, that 
NSF's current policies on conflict of interest are adequate 
to minimize bias in project selection that might derive 
from conflicts of interests. 


Merit Review and the Evaluation of the Extrinsic Utility 


of Research (See VI C, below) 


The Federal mission agencies have well tried procedures 
for selecting research work relating to their missions. NSF 
has been experimenting, with mixed success, over the past 15 
years with various mechanisms for relating research to nation- 
al needs. 


The Committee applauds NSF's current strategy of increasing 
research infrastructure investments in selected areas of na- 
tional need through merit review processes. Overlapping tech- 
nical (peer) and relevance panels, like the ones used in 
choosing NSF's Engineering Research Centers, provide a model 
review procedure for large projects and centers intended to 
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have an impact beyond the advancement of knowledge itself. 


Review of Inter- and Multi-disciplinary Research (IMR) 
Proposals (See VI D, below) 


To deal with IMR proposals no one has yet devised a sub- 
stitute for ad hoc groups of reviewers whose expertise spans 
the multiple perspectives and skills required. Panels are 
frequently a necessary part of the process, and individuals 
who have penetrated disciplinary boundaries can play a crucial 
role. Further, the Committee stresses the critical role of 
high quality program staff in this area. Particularly important 
is their openness to exploratory inguiries from the field, 
and their willingness to take the trouble to negotiate joint 
funding arrangements with other related programs. NSF's 
within- and cross-Directorate IMR programs provide models of 
how such proposals can effectively be handled. 


Merit Review and Improving the Infrastructure of 
Academic Science and Engineering (See VI E, below) 


a) The need for strong management commitment (See VI El, below) 


Infrastructure and capacity buiiding are long term concerns 
which require strong leadership and management commitment 

to framing and fulfilling long range plans. Top management 
must periodically reaffirm the goals and must ascertain that 
all the relevant review criteria for project selection are 
in fact being utilized. This requires the creation and use 
of information bases about the review process. 


b) Minorities, women and the handicapped (See VI E2, below) 


The effort to utilize all available science and engineering 
talent is a matter of national need as well as equity. Set- 
aside programs to bring underutilized groups into science 
and engineering serve an important transitional role. But 
the long term goal is for members of such groups to succeed 
in general competition. There is a danger that set-aside 
programs will fail to move collective capacities far enough, 
or to encourage individuals to achieve at a level consistent 
with their capacities. The Committee recommends that NSF 
aim to supersece set-aside programs for underutilized groups 
with a preferential strategy. At the same time, adequate 
information must be generated to track and assure applica- 
tion of appropriate review criteria, and use of appropriate 
mix of reviewers. 


c) Geographical distribution (See VI E3, below) 
After examining the available data the Committee was con- 


vinced that the geographical distribution of NSF research 
funds is roughly proportional to the current distribution 


d) 


e) 


of research related resources in the states and regions. 

To build national research capacity, however, areas which 
now have limited resources mus* be targeted for infra- 
structure improvement. The Committee applauds NSF's 

EPSCOR program (Experimental Program to Stimulate Competi- 
tive Research), and its merit review process, as an example 
of the kind of capacity-building effort which is likely 

to be successful in this and other areas of infrastructure. 


Baccalaureate Institutions (See VI E4, below) 


The role of non-doctoral institutions in identifying and 
nurturing science and engineering talent is critical. These 
schools graduate over one third of all bachelors degree 
recipients who go on to earn science and engineering 
doctoral degrees. They also provide a significant portion 
of the bachelors and master degree graduates in science 
and engineering who go directly into the labor force. 
Targeting and capacity building strategies should be 
vigorously pursued to strengthen this sector of academic 
science and engineering, and measures should be taken to 
assure appropriate weighting of review criteria in the 
awarding of grants. 


Selection Process for Awarding Academic Science and 
Engineering Facilities (See VI E5, below) 


The Committee is concerned about the increasing frequ- 
ency Of awards of funds for academic research facilities 
by direct Congressional action, independent of agency 
planning and review processes. It believes that, in the 
absence of regularly budgeted agency programs for facil- 
ities, unmet needs are driving universities to seek facil- 
ities from any available source, including directly through 
their Congressional representatives. 


The Committee believes the Federal government has a respon- 
sibility to share in the costs of research facilities, and 
believes that in the long run, programs to provide such 
Support are indispensable for the vitality of the Nation's 
science and engineering enterprise. By establishing such 
programs, Congress would authorize funding agencies to 
design appropriate merit review processes for facilities. 
The Committee recommends that NSF urge that the process of 
determining Federal reponsibilities in this area be rapidly 
completed. 


Until such programs are in effect, however, a process is 
needed through which members of Congress can obtain inde- 
pendent technical (peer) assessment of proposed science and 
engineering facilities which are intended to enhance the 
national infrastructure for academic research. Such advice 
needs to be obtained before the items are included in agency 
appropriations. 
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6. Program Manager Discretion (See VI F, below) 


The Committee wishes to stress that high quality program 
managers are critical to the operation of the merit review 
process. Good program managers are the key in several situa- 
tions covered in these recommendations: in managing technical 
(peer) review, especially in the case of multi- and interdis- 
ciplinary research proposals; in making appropriate arrangements 
for including research planning and infrastructure considerations 
in the review process; and in providing for the inclusion of 
relevance and utility criteria in the decision process. 


Furthermore, the ability of program managers to sense the 
potential, as weil as the current state, of their fields is 
important in the support of innovative research. The Committee 
recommends that there should be continued consideration of 
arrangements to permit program managers increased discretion 
over a portion of their budgets (5-10%) to support or renew 
support for creative and high risk research proposals. 


7. Accountability in the Review Process (See VI G, below) 


The need for accountability in the allocation of public 
resources creates tensions with the need to depend on the 
research community for confidential expert advice in the 
review process. This tension attracts bureaucratic solutions, 
including increases in required procedures, documentation, 
and rules on disclosure and audit. 


The Committee believes that increasing the overlay of 
bureaucratic procedures is no solution to the problems of 
accountability in the review and project selection process. 
What is needed instead are informality, flexibility, and the 
capacity to make subtle distinctions which stand on the 
reputations, credibility, and professional integrity of 
program officials and reviewers alike. The Committee affirms 
that this is the kind of response needed from funding agencies 
to support and propel the discovery process in basic science 
and engineering research. 


II. RECOMMENDATIONS 


The Committee recommends that the term “Merit Review" be 
adopted to refer to processes of review, rating and recom- 
mendation of proposed science and engineering research and 
education projects in which the selection criteria include 
not only technical excellence but also additional factors 
such as improving science and engineering infrastructure, 
or practical relevance of the research. 
(See Sections I-l and V) 


The Committee is concerned about the inadequacy of the 
data for tracking the processes of merit review at NSF. 
It recommends improvement of the data system to enable 
management to answer key questions about the functioning 
of the process, and to provide improved support for program 
managers in obtaining the best reviewers. 

(See Sections I-2a, VI-Bl and VI-G) 


The Committee encourages NSF to strive to meet its current 

goal of reducing proposal processing time to a maximum 

of six months from proposal submission to final decision. 
(See Sections I-2b and VI-B2) 


In the interest of improving the quality of mail reviews, 
the Committee recommends that NSF's instructions to review- 
ers be revised to include requests for full coverage of the 
proposal and for moderate language. Also recommended is 

the exploration of means to provide reviewers with feedback 
on their contributions, such as sending them the annual 
listing of awards in the program for which they have 
reviewed. (See Sections I-2c and VI-B3) 


The Committee recommends as a model for the review and 
selection of large science and engineering research and 
education projects and centers, intended to have an impact 
beyond the advancement of knowledge itself, the overlapping 
technica’? and relevance panels as used in NSF's Engineering 
Research Center program. 

(See Sections I-3 and VI-C) 


As models for handling the review and selection of inter- 
and multi-disciplinary (IMR) research proposals, the 
Committee recommends NSF's recently instituted within- 


Directorate (Engineering) and cross-Direc te (Chemistry 
of Life Processes) programs. 
(See Sections I-4 I=-D) 


7. With regard to programs designed to improve the infra- 


structure of U.S. science and engineering, the Committee 
recommends the adoption of long-term, preferential capacity 
building programs, affirmed by strong top management 
commitment, and backed by data bases about the merit 
review processes which can document equitable treatment of 
proposals. (See Sections I-5 and VI-E) 


The Committee recognizes that Congress has initiated steps 
to determine the extent of needs and the appropriate 
Federal role in the support of facilities for academic 
science and engineering research. The Committee recommends 
that the NSF urge that the process of determining Federal 
responsibilities in this area be rapidly completed. 

(See Sections I-5e and VI-E5) 


The Committee recommends that NSF program officers be 

encouraged to continue present efforts to support creative, 

high-risk research, and that they be given discretion over 

a portion of their budgets (e.g.,5-10%) to accomplish this. 
(See Sections I-6 and VI-F) 


Iii. INTRODUCTION 


From the earliest days of Federal support for science, 
research funding agencies have had to be selective in the award 
of funds. A major task of the program manager has therefore 
always been to search for excellence. To aid in that search, 
the program manager has traditionally used advisers, trusted for 
their competence and judgment about the scientific promise and 
technical merit of proposed work as well as the qualifications 
of the proposing investigator. In the early days of the National 
Science Foundation, this review process was quite fluid: the 
first set of NSF grants went from proposal to approval by the 
National Science Board in two months, including several levels 
of review. 


But times and circumstances change. The environment for 
peer review has shifted significantly, as described below in 
Section III. There are new pressures stemming from shifts in 
the structure of science and engineering research, and from 
changes in the social and political environment. Peer review 
is now being applied to a much wider range of selection situa- 
tions, sometimes with potential technology applications as well. 
At the extreme there are very large facilities which could by 
themselves consume half of the total Federal funds for academic 
research for a single year. The traditional concept of peer 
review, as assessment of scientific excellence of individual 
research projects by peers in the same or adjacent fields, is 
too simple to describe the current diversity and complexity 
of the project selection system. 


Technical review by peers remains, as it has been for four 
decades, the cornerstone of the project selection system. Under 
present circumstances, however, non-technical policy consider- 
ations often become attached to research funding decisions, 
especially when these involve program support or facilities. 
When this happens, the public, and the research community itself, 
may come to confuse the technical and non-technical aspects of 
funding decisions. The credibility of technical (that is, peer) 
review, and ultimately of science itself, is thereby weakened. 


The Committee found it important in its work to distinguish 
clearly between procedures for technical review and the means 
for incorporating other, non-technical considerations into the 
decision process. The Committee adopted the term "merit review" 
for describing both of these processes together. Section V 
below describes some existing merit review procedures, and 
Section VI discusses issues and appropriate procedures for some 
particularly important types of funding decisions. 


The Committee believes that the procedures used to select 
research projects in Federal Agencies are generally functioning 
well. Tight resources, of course, enhance strains in any system. 
Under the current funding situation many good proposals 
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remain unfunded, and it is widely believed that there is 
reduced willingness to fund risky proposals with high potential. 
But not all the discontent with the process of project selec- 
tion can be attributed to shortage of funds. Other complaints 
can and should be addressed with improvements. Some are 
suggested in Section VI B below. 


IV. CHANGES IN THE U.S. SCIENCE & ENGINEERING SYSTEM 


A. The Social and Political Environment of Science & Engineering 


The growth of U.S. R&D, the expansion of the academic re- 
search system, and the Federal role in these developments are 
quite well known. For the purposes of this report it is 
useful to distinguish four periods: 


o The spectacular contributions of science and technology to 
the war effort in the 1940's, and the astute activities of 
a few senior statesmen of science, led to a Federal commit- 
ment to assume major responsibility for basic research 
Support. The NIH and the NSF were established, and project 
selection by technical peer and peer panel review became 
a cornerstone of the system. 


o The rapid growth of higher education in the 1950's and 1960's 
greatly expanded academic research capacity, which in turn 
was fueled by fast growing Federal expenditures -- not only 
for project research, but also for research facilities and 
scientific and technical manpower growth programs. 


© From the late 1960's to the mid 1970's there was a slow-down 
in the rate of expansion of Federal funds for academic R&D 
(in constant dollars, there was a slight decline). New 
Federal R&D priorities arose in the form of new agencies 
and programs for applied civilian research in problems of 
energy supply, environmental pollution, and continued growth 
of health related research. Despite worries about the 
“steady state university", academic research communities 
continued to grow, albeit at a reduced rate. 


© Tne current decade has again seen significant shifts in 
Federal R&D priorities. The importance of basic research 
has been reaffirmed, but in a changed context which gives 
greater emphasis than before to its linkages with the 
development of key national needs for defense and for 
economic competitiveness. 


The linkages of the disciplines constituting basic research 
to technologies are manyfold and not always obvious. They can 
be likened to a fluid drive transmission where vigorous 
activity in one sphere is variably transmitted to another. 
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The relationship of new technoiogy to economic growth, on 

the other hand, may be more analogous to a mechanical system 
of cogs and gears. However this may be, concern with these 
issues has renewed the interest of policy makers in R&D 
program design and evaluation, and in project selection 
mechanisms. Technical (peer) review is seen as a necessary, 
but for some purposes, not sufficient component of the project 
selection process. 


In the current period, major pressures upon Federal 
resources require difficult allocation decisions. While 
academic R&D has fared comparatively well in recent Federal 
budgets, restrictions are coming at a time when academic 
science and engineering research is particularly vulnerable. 
The demographics of student enrollments for the rest of the 
century is not positive. Student interest in science majors 
at both undergraduate and graduate levels is not encouraging. 
Shortages of faculty in several areas of science and engineer- 
ing may bkecome exacerbated as retirements of the boom faculty 
of the mid-century pick up. Critical needs are expressed 
for capital investment in R&D facilities, in part due to 
deterioration and obsolescence of the capital stock built 
during the expansion period of the 1960's, and in part due 
to escalating costs of the “sophistication factor" in frontier 
research equipment and special facilities. In most fields 
it costs orders of magnitude more, in constant dollars, to 
operate at the frontiers of research today as compared with 
four decades ago. 


Another key pressure on basic science is the increasing 
demand that it serve other public functions. While the obli- 
gation of the Federal government to provide major support for 
basic research has become firmly established, the past decade 
has seen increasing concern among policy makers with improving 
the relevance of basic research to key national needs. This 
has, of course, generally been true for the basic research 
supported by Federal mission agencies. But it is important to 
recognize that it has also gradually become part of NSF's 
mission. 


1. How "Relevance" Criteria became part of NSF's Mission 


In 1968, in response to Administration and Congressional 
concerns about the acute needs for new technological solutions 
to many large and serious problems affecting society, NSF 
began a series of experimental programs aimed at improving 
the relevance of basic research to national needs. First, 
Congress amended NSF's enabling legislation to include “ap- 
plied research", in addition to "basic research" as an agency 
goal. NSF followed up in 1969 with a new program -- IRRPOS 
(Interdisciplinary Research Relevant to Problems of Our 
Society). This was replaced in 1971 by a much more comprehen- 
sive program -- the RANN (Research Applied to National Needs) 
Directorate which, in the language of NSF's budget proposal 
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for FY 1978, “focusses U.S. scientific and technical resources 
on selected problems of national importance for the purpose 
of contributing to their timely, practical solution." 


Other strategies were devised in the early and mid-1970's. 
Notable examples are the NSF's programs for industry/university 
cooperative research centers and industry/university cooperative 
research projects, and the small business innovation research 


program. By 1978, however, it was felt that the problem-focussed 


approach of RANN was inadequate, and that increased general 
support for research in engineering and applied science would 
better strengthen the nation's capacity to create new techno- 
logies. Accordingly, RANN was dissolved and in 1979 NSF's 
Engineering Division was raised to the status of a Directorate. 
This was done in part as an alternative to the proposed 
creation of a National Technology Foundation which would 

have removed Engineering from NSF entirely, and weakened the 
links between basic science and engineering. 


The most recent NSF response to improving the relevance of 
science and engineering research and education to national 
needs is the Engineering Research Centers (ERC) program. In 
contrast to the RANN approach to focus on specific selected 
problems and technologies, the ERC's represent an infrastruc- 
ture development approach, to provide resources and opportuni- 
ties for academic engineering to develop new bases for research 
and education in collaboration with industry. 


Not only has NSF created separate programs for these pur- 
poses, but in the mid-1970's it incorporated in its project 
selection process an explicit selection criterion which asks 
reviewers of proposals to consider, “the utility or relevance 
of the [proposed] research ... to the achievement of a goal 
that is extrinsic ... to that of the research field itself, 
and thereby [may] serve as the basis for new or improved 
technology or assist in the solution of social problems.” 


Of course, changing the criteria for selection without 
explicitly altering the composition of reviewer groups will 
not necessarily result in effective application of the utility 
criterion. Reviewers selected for their familiarity with 
technical aspects of a proposal are not necessarily best 
equipped to make judgments about the potential utility of the 
research, or its impact upon the infrastructure of the science 
and engineering system. Most of the NSF programs that have 
explicit utility and/or infrastructure goals in addition to 
research goals are also explicit about including reviewers 
with expertise or experience pertinent to those goals in their 
review processes. Engineering programs, for example, have long 
used industrial reviewers as a matter of course, and the 
project selection system in NSF's former RANN Directorate 
explicitly included representatives of potential “user commun- 
ities” in its review process. 
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B. Changes in the Structure of Science & Engineering 


Four key changes in the structure of U.S. science and engi- 
neering are creating pressures upon the peer review system 
for project selection. 


oOo The sheer growth in the size of the research community 

has forced standardization of proposal machinery for managing 
the system. This is most true for the agencies required to 
maintain an “open window" -- NIH and NSF -- where, with few 
exceptions, all proposals must undergo review. 


The growth of the academic research community during a 
period of declining (1967-1976),or more or less steady state, 
(1976-present) resources, coupled with escalating capital and 
equipment costs of research is putting severe competitive 
pressures on the project selection mechanism. Project selection 
processes suitable in periods of expansion are not necessarily 
optimal during periods of static or declining resources. 


Oo While convincing supportive evidence is hard to obtain, it 

is widely believed that science is increasingly moving from 
"little" to "bigger" science. Review of large project proposals, 
involving multiple investigators, large equipment or facilities 
and multi-year commitments, is considerably more complex 

than review of single-investigator, "small" projects. 

Large projects may involve significant policy issues concerning 
matters of research infrastructure (shifts in the distribution 
of research resources between regions, types of institutions 
and the like), as well as concerns about economic impacts, 
which go beyond purely technical considerations. 


The trend toward “bigger" and more collaborative or collec- 
tive research is being driven by two major structural features 
which themselves are linked: the increasing capital costs of 
research, and the increasingly inter- and multi-disciplinary 
character of much science and engineering work. 


o Frontier scientific instrumentation and facilities are 
moving targets -- they are constantly improving, opening up 
whole new areas of phenomena for investigation. Without access 
to these tools investigators are excluded from areas of study, 
and scientific progress is slowed. Complex, expensive research 
equipment and facilities may actually decrease the cost per 
unit of scientific information because of their enormously 
enhanced sensitivity or processing capabilities. Nevertheless, 
the price to stay at the leading edge of research appears to 
have been increasing exponentially. 


The construction cost of leading edge facilities in particle 
physics gives an idea of the problem: in 1950 the Berkeley 
Cyclotron cost about $10 million ($16M in 1972 constant 
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dollarz); in 1970 Fermilab cost about $100 million ($110 
million in 1972 dollars); and, if it is constructed in 1990, 
the cost of the Superconducting Supercollider (SSC) is esti- 
mated at about $4 billion (nearly $2 billion in 1972 constant 
dollars). In another example, a 1980 study by the Associat- 
ion of American Universities, The Scientific Instrumentation 
Needs of Research Universities, reported the actual start-up 
instrumentation costs for two new faculty members in synthetic 
organic chemistry in 1979 and 1979. Their personal laboratory 
equipment costs rose at a rate of 30% annually in constant 
1972 dollars, and the cost of the larger, more expensive, 
departmental equipment to which access was required went up 
36% per year in 1972 dollars. 


In both “big” and “little” science, then, decisions about the 
allocation of these resources become increasingly subject to 
policy and planning considerations, in addition to the crite- 
rion of scientific excellence. The review process must take 
this into account. 


o Another structural trend with serious implications for 
project selection is the emergence of new areas of investi- 
gation which have an inter- or multi-disciplinary character. 
We are seeing emerging aggregations into transdisciplinary 
and even transprofessional groupings such as “scientist/engi- 
neer", and “science/medicine/engineering”. Once again it 
becomes apparent that review and selection processes appro- 
priate to single investigators in single disciplines must be 
adapted to these more complex and changing circumstances. 


V. MERIT REVIEW PROCEDURES 


Although the procedures for utilizing expert opinions vary 
widely, the judgment of scientific excellence lies at the 
heart of the funding decision procedures of every research- 
sponsoring agency. But since scientific excellence is often 
a necessary but not fully sufficient condition for funding, 
the selection process also includes elements which bring in 
criteria other than technical excellence. Even in the early 
days of NSF, for instance, advisory panels assisted the pro- 
gram manager in selecting a portfolio of projects which 
reflected the broad interests of disciplinary development as 
well as the the specific criterion of excellence. At the 
National Institutes of Health, advisory councils (which 
consider the relevance of proposals to health goals) were 
established by legislation even before study groups (which 
judge purely scientific excellence) appeared. In programs 
with the goal of building scientific institutions, it is 
generally acknowledged that expertise and experience beyond 
the judgment of scientific excellence is needed to make the 
best funding decisions. 
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The Committee affirms the importance or the search for 
excellence as a necessary condition in the distribution of 
Federal research funds. To fail to obtain expert advice on 
research quality is to risk misallocation of public funds. 
However, the Committee also recognizes the importance of wider 
considerations which enter, often by legislative mandate, into 
funding decisions. 


In its hearings, the Committee gathered information about 
how merit review is structured in the various research-sup- 
porting agencies. Selected descriptions are presented in 
Appendix I. 


Within the National Science Foundation merit review takes 
several forms. 


First, considerations that go beyond scientific excellence 
are included in the four criteria by which all proposals to 
NSF are assessed. These four criteria are: 


1. Research performance competence. This criterion relates 
to the capability of the investigator, the technical soundness 
of the proposed approach, and the institutional resources 
available to the investigator. 


2. Intrinsic merit of the research. This criterion assesses 
the likelihood that the research will lead to new discoveries 
or fundamental advances within its field of science or engi- 
neering, or have substantial impact on progress in that field 
or in other scientific and engineering fields. 


3. Utility or relevance of the research. This criterion is 
used to assess the likelihood that the research can contribute 
to the achievement of a goal that is extrinsic, or in addition 
to, that of the research field itself, and thereby serve as the 
basis for new or improved technology or to assist in the solution 
of social problems. 


4. Effects of the research on the infrastruct‘ire of science 
and engineering. This criterion relates to the potential of the 
proposed research to contribute to better understanding or 
improvement of the quality, distribution, or effectiveness of 
the Nation's scientific and engineering research, education and 
manpower base. 


All four of these criteria are taken into congideration in the 
proposal award decision at all levels -- individual mail 
review, panel discussion, program manager recommendation, and 
further management review. The relative weighting of the 
criteria in the judgments of the reviewers and program managers 
varies according to the type of project or program. 


Second, special steps are sometimes added to the review 
process to consider particular criteria. In the Engineering 
Research Centers Program, for instance, first stage peer 
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panels judge the technical excellence of a proposed center and 
the impact of its educational plan and its plan to involve 
industry. Then a more broadly based panel is convened to 
review the best of the proposals for technical and managerial 
excellence and the importance of their topics to national 
productivity. Technical excellence and the importance of the 
topic for technological competitiveness are paramount in the 
deliberations; geographic distribution and impact on engineer- 
ing infrastructure are additional criteria. To give another 
example, when a mathematical research institute was proposed, 
broad discussion was stimulated within the mathematics commu- 
nity on the best allocation of funds for support of the field. 
This discussion was taken into consideration explicitly in 

the final funding recommendation. In the case of Regional 
Instrumentation Centers in chemistry, a separate step in the 
NSF review process explicitly considered the question of 
geographic distribution 


Third, particular emphasis on some of these criteria is 
built into the structure of certain Foundation programs. For 
instance, in the Visiting Professorships for Women program a 
panei is convened by a cross-directorate program. This panel 
discusses mail reviews (which have been solicited by the 
research program managers) and gives special consideration to 
the non-research elements in recommending final award decisions. 
Similarly, funding decisions on projects supported under NSF's 
International Program are recommended by special program 
officers, who can weigh the judgments of scientific excellence 
provided by reviewers against diplomatic considerations. 


These varied patterns for incorporating criteria other than 
excellence into project selections are reflected in other 
research-supporting agencies as well. Sometimes relevance and 
excellence criteria are used together, as in most NSF decisions. 
At «he Office of Naval Rerearch, for example, relevance of 
research to long-term Navy objectives is an important consider- 
ation in funding. These judgments are built into ONR's programs 
in several ways. The distribution of funds among research areas 
is determined in part through the seiection of Accelerated 
Research Initiatives (ARI's), which are rated on scientific 
promise and Navy relevance separately, but which are ultimately 
selected with both criteria in mind. At a later stage, ONR 
scientific officers select individual projects on the basis 
of excellence as well as relevance. 


Sometimes the two criteria are clearly separated in the 
review process. At NIH, for instance, study sections organized 
on a disciplinary basis rate proposals on quality. Then, 
Institute Councils consider the implications of the proposals 
for health problems, sometimes departing from the funding 
priorities the study sections recommend. Similarly, at the 
Department of Energy, when major research facilities are 
built, technical review and site selection are treate as 
distinct steps in the funding procedure. In the case of the 
Nationa). Accelerator Laboratory (Fermilab), for instance, 
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more than 100 proposals were received by the Atomic Energy 
Commission. The Commission screened the sites to insure that 
they met basic requirements, including adequate power and 
water supplies, adequate land areas and enough housing and 
transportation capacity. Once these determinations had been 
made, a committee of experts assembled by the National Academy 
of Sciences further reduced the number of proposals to a short 
list of sites that met all the requirements for a successful 
national laboratory. The Commission made the final selection, 
taking into account all the relevant social, economic and 
political priorities. 


In other facilities programs, however, special panels have 
used mixed criteria in making specific funding recommendations. 
One example is the Health Research Facilities Fund, operative 
at NIH in the 1950's and 1960's. This program used panels 
which were more diverse than the usual NIH disciplinary study 
sections. Their members were drawn from several areas of 
science and included university and hospital administrators. 
These groups were further advised by a strong staff of archi- 
tects and engineers. Thus several types of criteria entered 
into their allocation decisions. 


In summary, the Committee found no instance of project 
selection in which the criterion of scientific excellence was 
ignored by a funding agency. In single-discipline, single- 
investigator programs this criterion clearly dominates the 
decision process. Yet other criteria are frequently incorpo- 
rated as well through a wide range of mechanisms. Clearly, 
merit review -- which includes, but is not limited to, con- 
siderations of scientific quality -- is a ubiquitous feature 
of the current Federal research funding system. 


VI. FINDINGS AND DISCUSSION 


A. Introduction and Overall Assessment 


Merit review procedures are constantly evolving in response 
to the changing requirements of science and engineering 
and their environments. The Committee is impressed with the 
inventiveness of the NSF and other agencies in creating pro- 
cedures to protect the criterion of technical excellence 
while incorporating additional considerations into their 
funding decisions. 


The Committee finds that the traditional technical peer 
review process is working well. Although it noted some 
complaints and dissatisfaction with the system, the Committee 
believes that a significant portion of these complaints can 
be traced to financial pressures on the system, which result 
in declination of many worthy proposals. Other complaints 
can and should be addressed with improvements: some are 
suggested below in Section B. 
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Sections C,D and E discuss the complexities of some of the 
situations which call for the utilization of criteria beyond 
scientific excellence in the selection process. These include 
several special types of programs, such as those which aim 
to improve aspects of the infrastructure of science and 
engineering; those which evaluate the potential extrinsic 
utility of research results; and those which support inter- 
and multi-disciplinary research. The final two sections (F 
and G) examine the role of the program manager and the issue 
of accountability in the review and selection process. 


B. Improving Process and Quality of Review 


1. The Reviewer Pool 


Questions about the openness, characteristics and utili- 
zation of the reviewer pool are both sensitive in our demo- 
cratic society, which tends to be suspicious of elitism and 
in-group monopolies, and difficult to address because of the 
lack of data. The several special studies (See Appendix IV) 
of these issues over the past decade confirmed that there is 
no evident pattern of discrimination by reviewers in terms 
of institutional affiliation, age or gender. 


How reviewers get into the system, how they are utilized 
and how long they stay in, are all questions that can only 
be addressed anecdotally at present. It is also not known 
how often individual reviewers are asked to work, nor how 
often they actually submit responses. The Committee recom- 
mends a serious effort to develop data to address these 
questions. 


It is known that NSF has a reviewer pool with about 
150,000 names; that about one-third of them are asked to 
provide a review in a given year; that about one quarter of 
them actually provide at least one review during a given 
fiscal year; that about 125,000 mail and panel reviews are 
conducted each year; and that about 15,000 new reviewer 
names are added each year. These numbers rebut the criticism 
that peer reviewers are exclusive in-groups. They suggest 
but do not prove an open system. 


Names of new reviewers come from several sources. At NSF 
each review form contains a space for “additional suggested 
reviewers." Also program managers ask for suggestions at 
professional meetings. One suggestion debated without con- 
clusion by the Committee was to ask proposers to nominate 
possible reviewers for their work. Some NSF programs already 
make these requests. A number of professional journals 
routinely request this of their authors, and several instances 
were recounted where author-suggested reviewers were tougher 
on the manuscripts than other reviewers. The recruitment of 


new reviewers at NSF is probebiy , be! ped by NSF's policy of 
having a sizeable proportion of its program managers employed 
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as "“rotators" for periods of one to three years. Most of these 
come from and return to university research. Being close to 
the action they bring fresh perspectives to the programs 

as well as new sets of reviewers. 


Concern was expressed about reviewer response rates. The 
Committee heard complaints that in some cases 6-7 requests 
for reviews must be mailed in order to receive three. This 
level of response could "unbalance" sets of reviewers which 
had been custom designed for particular projects. However, 
despite specific probing of agency representatives, the 
Committee received no evidence of any trend toward reduced 
response rates. Nevertheless, means to improve or maintain 
reviewer response rates should be sought. Better feedback to 
reviewers, if not too costly, would be a valuable courtesy 
to those who serve as reviewers. For example, all reviewers 
used by a program in a given year could receive the program's 
award list at the end of the year. 


The willingness of thousands of scientists and engineers 
to spend hours of concentrated thought on providing reviews 
without compensation may seem mysterious to many. Reviewing 
has several compensations, however. There is a certain pres- 
tige and recognition to being selected as a reviewer by a 
major research-supporting agency. Reviewing is also highly 
educational. It provides a window on forefront ideas in the 
area Of ones specialty. It also educates the reviewer in the 
format and layout of preparing good (and bad) grant proposals. 
These are important reasons for broadening participation in 
the reviewer pool. 


Much more information is available about the number and 
characteristics of scientific advisory panels. (See Appendix 
V). Their memberships can be, and are, tailored to the functions 
they perform. They are publicly visible and the fairness and 
appropriateness of their compositions can be publicly judged. 


2. Promptness 


The Committee heard evidence of dissatisfaction in this 
area, especially in the context of exploiting innovative ideas. 
A nine to twelve month review and award horizon is not con- 
ducive to innovative research. The Committee's recommendation 
under VI-F to provide program managers with some discretionary 
funds for innovative and creative work would provide "fast 
track" capability for processing at least some of these ideas. 
ideas. In addition, several administrative process improvements 
are under way at NSF which aim to bring the maximum processing 
time down to six months. 


It is especially necessary to seek these efficiencies 
if greater usage of merit review processes is anticipated. 
Two stage proposal processing for large projects and projects 
with wider considerations than the science itself, will tend 
to increase average proposal processing time. 
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3. Quality of mail reviews 


The Committee heard anecdotes about dissatisfaction with 
the quality of mail reviews -- perfunctory remarks at one end 
of the spectrum, and sweeping, sometimes vitriolic critiques, 
at the other. While it has no reason to believe these extremes 
are widespread, the Committee recommends that serious study 
should be made of the extent of these behaviors. The strength 
of the review process can be -no greater than the willingness 
of the community to participate in it with vigor and integrity. 


The issue of whether the names of mail reviewers should be 
revealed to proposers was raised, but not supported. A com- 
plainant suggested that reviewers would be more responsible 
if their names were attached to the reviews. The counter 
argument was that signed reviews would fatally dilute the 
criticism essential to a vital review process. Reviews would 
become more like letters of recommendation. NSF currently 
publishes listings of recently utilized mail reviewers at 
the Divisional level, thus making possible a broad assessment 
of who is in the pool. However, this does not force a sense 
of responsibility upon reviewers who may not already have one. 


At NSF, as a matter of policy, individual reviews are 
automatically returned to the proposer; if necessary, they 
are edited for anonymity by the program manager, but this 
does not necessarily mean that strong language is excised. 

It is felt that this practice has an important educational 
value in improving future proposals. It also provides grounds 
for complaints to those who feel their work has received 
inadequate or biased review. 


Further, for a number of years NSF has had a formal recon- 
sideration process in place which provides for examination of 
the technical correctness of the review process. It is 
regularly, but not frequently invoked. (See Appendix VI).The 
Committee considered in detail one case submitted by a com- 
plainant, and concluded that while several of the specific 
complaints clearly had substance, the process of review and 
appeal had worked fairly and well. The reviewer set had 
included both relatively perfunctory and sweeping, one-sided 
reviews, but taken as a set, there was a remarkable consistency 
among their substantive critiques. NSF has sent Special 
Notices to universities calling their attention to the recon- 
sideration process, but the Committee suggests that NSF 
explore additional means for advertising the process among 
proposers. 


In sum, given the voluntary and uncompensated nature of 
the mail review system, the Committee expressed doubts about 
influencing reviewer performance. Experiments could be con- 
ductea inviting proposers to rate reviewers remarks, which 
could then be transmitted to the reviewers. Letters of in- 
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struction to reviewers could be revised to include requests 
for full coverage and moderate language. Perhaps most impor- 
tant is to provide program managers with the support and the 
data about reviewers they need to make the best judgments 
they can in selecting responsible reviewers. 


4. Panel versus Ad Hoc Mail Peer Review 


Non-scientists, and even scientists and engineers them- 
selves, find the parallel existence of two basic forms of 
peer project review puzzling. 


Ad hoc mail peer review is the predominant form used in 
the physical and mathematical sciences at NSF. Program man- 
agers select a "custom" set of reviewers for each proposal 
to provide a balanced view of the various aspects of the 
work. The reviews are advisory to the program manager, who 
may Or may not take the advice. The studies (See Appendix IV) 
show, however, that there is a very high correlation between 
average reviewer scores on proposals and the decision to 
fund. As described above, the names of the reviewers are 
removed from the verbatim mail reviews and the reviews are 
automatically sent to the proposer. The program manager in 
these disciplines writes analyses of the reviews, and relates 
them to his recommendation to the Division Director. These 
documents become part of the record, and are likely to be period- 
ically examined in the triennial Program Oversight Review 
process (See Appendix II). This system is obviously highly 
dependent upon competent and committed program managers, 
responsible Division Directors, and alert and active Oversight 
Review Committees. 


Peer panels by themselves, or in combination with mail 
reviews, are prevalent in the biological and social sciences. 
NIH uses panels and study sections exclusively. NSF programs 
in these areas generally supplement panel discussion with 
mail reviews. The panel process differs from mail review in 
that it is analogous to a jury proceeding. Panels permit 
airing of differences of opinion, and if appropriately 
balanced, can produce more “representative" decisions than 
mail reviewers operating in isolation. Also, the interaction 
may produce new ideas, even though this may not necessarily 
benefit the proposer under discussion. On the other hand, 
panels may become dominated by one or two powerful personal- 
ities, thus tipping the balance in particular directions. 
Good program managers learn how to handle these cases. 


A difference between the NSF and the NIH panel systems 
is that the proposer in the latter receives only a summary 
of the panel deliberations on the proposal which may be less 
informative, if less painful, than verbatim mail reviews. 
NSF programs in biological and social science return to the 
proposer both the panel summary, supplemented by explanatory 
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notes from the program managers, and the verbatim mail reviews. 


The rhythms and timetables of the panel procedure differ 
from ad hoc mail review. Proposals destined for a panel pro- 
cess must follow the panel meeting schedule, while the mail 
process can respond more immediately if funding is available. 
Also, the costs of the two systems are distributed differently. 
Panels cost more in up-front agency travel and subsistence 
funds. The “costs" of mail reviewer time and effort are real 
and widely distributed, but they are not visibly charged to 
anyone's particular budgets. 


Another important difference is that panels tend to shade 
into some aspects of merit review. They can be more sensitive 
to infrastructural issues, and questions of balance and over- 
all direction ina field. 


In sum, the Committee recognizes that each basic type of 
review is rooted in the traditions of different research 
communities, and that each has certain strengths and vulner- 
abilities. No strong preference was expressed for either 
model. Perhaps, like corn, well planned and nurtured hybrid 
varieties promise the most vigorous yields for the future. 


5. Conflict of Interests 


In judging one's peers there is no impartial expert. Yet 
none other than peers are competent to make technical judg- 
ments. This is an inherent dilemma in scientific research. 
There is great reliance on an informal honor system, as well 
as awareness of the "golden rule" -- what you do to others, 
may be done unto you. At NSF these strong community norms 
are codified into a "Conflict of Interest" policy, a statement 
of which is sent to all reviewers. Reviews with a conflict 
of interest are not used in the evaluation process. In panel 
deliberations, panel members with a conflict of interest on 
a particular proposal must leave the room during the discussion 
of the proposal in question. 


In sum, whiie concern was expressed that the sense of honor 
might become frayed during periods of greater competition for 
tight resources, there are functioning policies in place to 
minimize bias deriving from conflicts of interests. 


C. Merit Review and the Evaluation of the Extrinsic Utility 
of Research 


Section IV A-l documented the increasing requirements on 
NSF to link basic research with a variety of national needs, 
including most recently, the requirements for economic compet- 
itiveness. 
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It must be understood that these developments, at least 
since 1980, have been taking place in a context in which a 
fundamental Federal policy of "privatization" has greatly 
reduced public support for technology development work held 
to belong in the private sector. In budgetary terms, there 
have been major reductions in Federal support for civilian 
applied research and development during this period. In a 
real sense, a greater responsibility for relevance to appli- 
cations has been laid upon basic research. Today, a greater 
burden is placed upon academic basic research support to find 
ways to improve the contributions of academic training and 
research to these needs, without compromising the freedom 
of academic research. 


NSF has responded with several special purpose programs, 
such as Small Business Innovation Research and University- 
Industry Cooperative Research Projects. New project research 
programs have been initiated to support work in areas previ- 
ously not seen as high priority, but now deemed to be valuable 
to improving economic competitiveness in such areas as design, 
processing and manufacturing engineering. In addition, NSF 
has chosen a strategy of creating focussed research centers 
on campuses to pursue research and training in areas related 
to national needs. The early version of this approach -- the 
Industry/ University Cooperative Research Centers program -- 
relied upon the financial and advisory participation of corpor- 
ations as evidence of the economic relevance of the proposed 
work, while obtaining traditional peer review of the scientific 
and technical promise of the proposed work. In this way, the 
Federal agency did not set research priorities. 


NSF's recent Engineering Research Centers program moves 
further toward including needs for economic competitiveness 

in the review process itself. As mentioned in Section IV, 
these centers have a two stage review process with two par- 
tially overlapping panels. The first stage reviews the scien- 
tific and technical excellence of the proposed work, the second 
reviews the subjects covered in relation to future needs. The 
Committee applauds this mode of overlapping technical (peer) 
and utility panels (together constituting a merit review 
process), and generally recommends the approach for large 
projects and centers intended to have an impact beyond the 
advancement of knowledge. The selection and meshing of the 
work of these panels is critical, and requires much attention 
by program managers. 


D. Review of Proposals for Inter- and Multidisciplinary 
Research (IMR) 


Review and funding IMR is a knotty problem for both mail 
and panel review systems. The majority of NSF's programs are 
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discipline-based, but often the most important and exciting 
work is going on at the interfaces of disciplines. To deal 
with multi- or interdisciplinary research proposals there is 
probably no substitute for specially constituted groupings 
of reviewers (utilizing both individual mail reviewers and 

a panel meeting and discussion) whose expertise spans the 
multiple perspectives and skills required. 


The Committee applauds the explicit procedures described 
by the NSF Engineering Directorate where there is an expec- 
tation that a program manager receiving a proposal only partly 
in his or her field, will seek names of additional reviewers 
from a program manager in the related area. If the reviews 
are favorable, joint funding between programs can be arranged. 
The relatively few “no-home" proposals are sent to a Director- 
ate-wide staff committee primarily set up to receive risky, 
creative and innovative proposals from program managers. If 
necessary, this group arranges with a program director for 
review of the proposal, and if the reviews are favorable, 
provides half of the funds from its budget, with the other 
half being from a related program if such a program can be 
identified. 


Where a multi- or interdisciplinary area of investigation 
appears to be coalescing, a separate program can be set up. 
At NSF, for example, the Chemistry of Life Processes program 
represents a cooperative effort between the Directorate 
for Mathematical and Physical Sciences, and the Directorate 
for Biological, Behavioral and Social Sciences. In this 
program proposals are reviewed in a two-step process. The 
relevant disciplinary programs conduct mail reviews. The 
results of these are fed into a final discussion by a multi- 
or interdisciplinary panel. 


On the whole in this area, the Committee stressed the 
role of strong, high-quality program staff. Particularly 
important is their openness to exploratory inquiries from 
the field. 


E. Merit Review and Improving the Infrastructure of Academic 
Science and Engineering 


1. The need for strong management commitment 


Building capacity for science and engineering research 
requires fundamental legislative and managerial leadership 
and commitment to framing and fulfilling long range plans 
and programs. Maintaining momentum is often difficult under 
pressures from more immediate objectives. Thus, top manage- 
ment must periodically reaffirm the goals, and subject the 
programs to searching scrutiny for progress toward the 
goals. This, in turn, requires data bases adequate to the 
tasks of tracking the developments in given areas of capacity 
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building, and for ascertaining that all the relevant criteria 
for review and project selection are being utilized. 


2. Minorities, women and the handicapped 


In discussions of improving participation of disadvantaged 
groups in science and engineering there is often an implicit 
assumption of an antithesis between peer review with integrity 
and a cluster of sensitive equity issues. In some agencies 
requirements for equity consideration are regarded as a man- 
agerial burden, or interpreted as if they are in collision 
with peer review processes. This Committee believes that the 
latter is not necessarily the case. 


The issue needs consideration, however, because if the sys- 
tem can be made more responsive, there will be less need and 
pressure for Congress to insert “set-aside” programs into 
agency budgets. These special purpose programs provide shel- 
tered competitions with access limited to certain classes of 
individuals or institutions. They do have important transi- 
tional value in that they can build an explicit consciousness 
and sense of responsibility into the organization concerning 
the needs and characteristics of the communities in question. 
But these programs tend to be seen as an “entitlement"™ strat- 
egy, which has a negative connotation. Agencies should aim to 
supersede them with a “preferential” strategy which should 
reflect a maturing of consciousness within the organization. 


A preferential strategy can be seen in terms of building 
capacity. Planned targets can be devised to increase the 
capabilities of certain kinds of institutions and/or comnun- 
ities over a given period. This is especially important in 
a period when there are many more excellent proposals than 
can be funded. For example, with two proposals of substanti- 
ally equal quality, the policy could be to fund one from 
the targeted type of institution, minority group, or gender. 
(An analogue might be in applications for G.I.loans -- extra 
points are given for certain characteristics. Of course, 
the first and fundamental sc 2ening must be for scientific 
excellence, and the extra points for special merit consider- 
ation should come afterwards.) 


A targeting approach differs in its long-run effects from 
a built-in, set-aside program for a special group. The latter 
does not encourage quality in the long run, because it kills 
important incentives for the designated beneficiaries. Set- 
asides also encourage another political process which attempts 
to limit the number of “entitled claimants", which otherwise 
tend to mushroom. 


Another effect of special limited competition programs is 
that they are essentially "plateau" or "ceiling" strategies. 
They do not move the institutional or collective capacities 
beyond the plateau into the terrain of ability to operate in 


BS 


-26- 


the general competition. The minority research community is 
somewhat split on this issue. Some are convinced that set- 
asides are bad, that they virtually label an awardee as some- 
one unable to compete in mainstream science. In fact, some 
have complained that some NSF Directorates were automatically 
referring their proposals over to the set-aside pot when 
they saw the name of the minority institution. Other minority 
researchers are not as concerned with the mechanism by which 
they receive their research support. 


When moving from special programs to a targeting strategy, 
problems may arise with accountability. The set-aside is a 
budget line item which is carefully nurtured and monitored 
by the target community. It believes it knows that all the 
earmarked support is being spent in the target community, even 
though questions about quality may be raised. The targeting 
approach moves away from this criticism, but it is also much 
harder to monitor. It requires more target community trust 
in the intent and goodwill of the agency, and greater agency 
management commitment to tracking and demonstrating the flow 
of proposals, review and awards. 


In sum, the Committee strongly recommends that agencies 
make efforts to move toward capacity building and targeting 
strategies, with strong program management, and serious 
periodic evaluation of the rate of progress. It must be added 
that capacity building is a long term process requiring con- 
sistent commitment and leadership from top agency managers. 


3. Geographical distribution 


The Committee examined data presented by NSF on the cor- 
relates of geographical distribution of Federal R&D funds 
and agency awards. On the whole, there appeared to be high 
correlations between NSF R&D funding to universities and 
colleges and data on various state characteristics -- for 
example, per capita state populations, and the number of 
academic scientists and engineers in each state. (See Appendix 
III for selected data.) 


The Committee applauds NSF's EPSCOR program (Experimental 
Program to Stimulate Competitive Research) which assists states 
in which research resources have been limited, to develop 
their capacities to compete for Federal research funds. EPSCOR 
is an exemplary case of a capacity building approach, with a 
well crafted merit review process for project selection. In 
general, however, the Committee saw nothing alarming in the 
pattern of distribution of research funds. In fact, seen 
over recent decades the degree of concentration of funds in 
academic institutions has even shown a slight decrease. 


4. Baccalaureate Institutions 


Over the decades Federal R&D policies have paid little 
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direct attention to the non-doctoral sector of academia. 
Federal science and engineering manpower development programs 
have focussed primarily at the graduate levels. In recent 
years federal policy makers have been concerned with improve- 
ment of precollege science and mathematics education. 


The role of baccalaureate institutions in identifying and 
nurturing science and engineering talent is critical because 
these schools graduate over one third of all baccalaureate 
degree recipients who go on to earn science and engineering 
doctoral degrees. Also, these schools provide a significant 
proportion of the bachelors and masters degree graduates in 
science and engineering who enter directly into the labor 
force. 


The NSF programs to support research projects and research 
equipment awards in these institutions are good examples of 
a capacity building and targeting strategy. Program manager 
initiative is especially important to stimulate proposals 


from the institutions, and to provide liaison with the research 


programs which conduct the technical review process. 


5. Selection Process for Academic Science and Engineering 
Research Facilities 


One reason for the formation of this Committee was concern 
with the absence of technical review of academic science 
and engineering facilities projects introduced into Federal 
Agency appropriations bills by means of Congressional 
"floor amendments." 


The members of the Committee are unanimous in their con- 
viction that well thought out programs for Federal contri- 
butions to building and refurbishing academic research facil- 
ities are indispensable for the vitality of the Nation's 
science and engineering enterprise. Further, the Committee 
believes that given the widely noted need for renewal of 
academic research facilities, the absence of any regularly 
budgeted Federal agency programs for facilities support is 
driving institutions to seek such funds wherever they can 
find them, including directly from their Congressional rep- 
resentatives. 


No one questions the right of the U.S. Congress to enter- 
tain petitions and to provide support for such projects as 
it sees fit. Large projects for research facilities may often 
have important implications beyond their scientific or engi- 


neering merit -- such as impact upon local or regional economies 
and employment. Nevertheless, the funding of science facilities 


without systematic technical review can be likened to building 


a dam without the assurance that there will be water behind it. 


Obviously, facilities support and project support are closely 
linked -- empty or underutilized facilities waste resources, 
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as do projects conducted in inadequate facilities. Planning 
is essential. 


A good illustration, which could be applied to other fields, 
is the nation's oceanographic research fleet. A balance of 
ships with different capabilities, based in east, gulf and 
west coast ports must be arrived at, phased into supporting 
agency budgets to allow for replacement and repairs, and 
carefully integrated with a longer range research plan. The 
planning process must include all the interested technical 
parties. Adding ships to the fleet outside the plan can 
cause dislocation and waste of resources. Such difficulties 
are likely to be compounded if no independent technical review 
is conducted. These matters are of special concern during a 
period when the priority needs of science and engineering 
cannot be fully supported because of budgetary stringencies. 


Congress is, of course, aware of these issues, and it has 
taken a number of recent initiatives to identify needs and to 
explore policies and programs in the area of academic research 
facilities. Bills have been introduced and debated, hearings 
have been held, and the creation of a sound data base has been 
ordered. The Committee recommends that NSF urge that the 
process of determining Federal responsibilities in this area 
be rapidly completed. 


Until regularly budgeted agency programs for facilities 
support are in place, however, the Committee believes that 
the Congress should explore setting up a process for obtain- 
ing independent technical assessment of proposed academic ; 
facilities projects prior to including the items in agency 
budgets. For example, it could set up an Academic Science 
and Engineering Facilities Authorization process which would 
funnel all the requests through a single path. A special 
group could be set up in the Federal Coordinating Council for 
Science, Engineering and Technology (FCCSET) to coordinate 
reviews, which could primarily be conducted by the executive 
branch agency envisaged as providing the budget. 


Under such an arrangement the Congress, of course, remains 
free to set its own priorities, but it would at least have 
the benefit of an independent finding about the scientific or 
engineering potential of the project. An analogue is the 
voluntary process whereby the President requests and is pro- 
vided with a technical assessment of the qualifications of 
proposed candidates for the Federal’ judiciary. No one is 
bound by the findings, but they make for better informed 
decisions. 


F. Program Manager Discretion 


The qualifications and commitment of the program managers 
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form a critical element in any system of project review. The 
program director is responsible for generating and properly 
using the flow of advice and opinion on his program projects, 
as well as addressing new opportunities in his or her area. 


The program manager must find a proper balance between 
running the review machinery and keeping it accountable, and 
acting as a stimulant and catalyst for new opportunities 
by encouraging proposals from the best in the research com- 
munity. The greater the overlay of inflexible procedures and 
requirements, the less attention can be devoted to issues of 
substance. Too much bureaucracy also reduces the likelihood 
of attracting and retaining high quality, creative personnel. 


The need for program manager discretion and innovation 
becomes more important if, as is widely believed despite the 
absence of confirmatory studies, the peer review process 
tends to have a centrist bias -- especially in times of 
tight resources. Peer review is believed to function as a 
"modulation" system which effectively cuts out the low end 
of the proposal quality distribution, but may also chop off 
part of the top end of the curve -- work seen as obviously 
creative, but non-traditional or highly risky. A competent 
program manager would want to have a (small) portion of the 
program funds invested in high-potential but high-risk work. 


In the light of these considerations, the Committee was 
favorably impressed with a new experimental procedure in the 
NSF Engineering Directorate which permits program managers, 
on their own initiative, to expend up to 5% of their program 
budgets on certain kinds of innovative small projects. These 
proposals are subject to internal review but are not required 
to undergo external peer review. 


A similar practice, but for renewal proposals, was initiated 
several years ago by the NSF Chemistry Division, and has 
since been extended to all of NSF. In this procedure (Creati- 
vity Awards) program managers can commit up to 10% of their 
program budgets to renew or extend exceptionally promising 
pieces of work without new external peer review. Both of 
these practices reward and encourage creativity. They also 
significantly cut down on paper work and provide a "fast 
track” mechanism to take advantage of opportunities. 


The program manager also plays an important role in pro- 
ject selection in some NSF programs which use a preliminary 
proposal process, e.g., Informal Science Education, and the 
Program for the Analysis of Science and Technology Resources. 
In these programs brief informal preliminary proposals are 
required or encouraged. They are reviewed by staff for promise 
and relevance to program goals, and on the basis of this 
review, preparation of a formal proposal is either encouraged 


or discouraged. A formal proposal may be submitted irrespective 


of the staff recommendation, although in most cases the staff 
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recommendation is followed. The staff recommendation on the 
pre-proposal has no bearing on the technical review process 
once a formal proposal is submitted. 


The burden of accountability for activities initiated by 
program managers rests upon post-performence evaluation by 
superiors and outside program review groups. In general, as 
one reduces the burden of prior technical review (on the 
grounds that the program manager is a responsible technical 
judge) one must increase post-performance evaluation activities. 


NSF policy requires that each research program be subjected 
to a thorough external review of substance and process once 
every three years. This Program Oversight Review process is 
conducted by outside panels, frequently sub-groups of Division- 
al Advisory Committees. The oversight groups look at samples 
of grant documentation for both awards and declinations, and 
they assess the fairness and scientific balance of the program 
manager's judgments. The groups‘ reports are made available 
to upper management and the NSF Director, who, in turn, makes 
an annual report on the oversight review process to the Nation- 
al Science Board (See Appendix II). 


While acknowledging the importance of letting creative and 
non-traditional ideas in “at the front door,” it is also 
recognized that the research system is full of “back doors.” 
The very nature of scientific research is uncertainty. Thus, 
the connection between the content of proposals for research 
and the working out of the research process cannot and should 
not necessarily be direct. Abuse of the obvious need for 
flexibility, i.e., non-performance, as opposed to good-faith 
pursuit of alternative lines of work, can be partially con- 
trolled by alert and knowledgeable program managers. In addition, 
NSF has recently instituted a requirement in its review pro- 
cess that asks investigators to include descriptions of prior 
NSF supported work in their proposals. Reviewers are requested 
to record their judgments about past performance. 


The role of the program manager is especially critical in 
special-purpose programs, such as small business innovative 
research, special programs for minorities and women, for 
small colleges, and for research capacity building in “have- 
not" states. Special knowledge of, and intensive interaction 
with, the target communities require much program manager 
time and initiative. 


G. Accountability in the Review Process 


A hallmark of scientific research is its insistence upon 
open publication of work which is freely accessible by the 
whole community. This rigorous accountability is enforced by 
community vigilance on questions of priority in discovery. 
This openness is apparently contradicted by the anonymity of 
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merit review processes, whether in competition for awards or 
research support, or for publication of work in scientific 
journals. It is a doubly difficult matter to comprehend when 
public funds are involved. Merit review as practiced in 
Federal agencies remains advisory to the agencies, thus 
keeping the public officials responsible for decisions. But 
probably in no area of public endeavour other than science 
are public officials permitted to keep the identities of at 
least some of their advisors confidential. 


This situation makes the system peculiarly vulnerable to 
the criticism that in-groups may exercise undue influence 
upon decision-making. Critics propose as remedies a variety 
of set procedures, documentation, and requirements for 
disclosure and auditability, in sum, more bureaucracy. Yet 
a major theme that pervaded the presentations and discussions 
before this Committee, was the need for informality and 
flexibility, and the capacity to make subtle distinctions 
which themselves may be disputable, but which nevertheless 
stand on the reputations, credibility and professional integ- 
rity of program officials and reviewers alike. These are the 
qualities needed to support and propel the discovery process 


in basic science. 


On the grounds that increased accountability can be obtain- 
ed by providing information about how a system works, the 
Committee strongly supports the creation and maintenance of 
a significantly improved data base about the review process. 
(See the recommendation in Section VI Bl). Improved data 
should benefit both the research managers themselves as well 
as wider public understanding of the advice systen. 
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APPENDIX I 


PEER REVIEW IN SELECTED FEDERAL AGENCIES 


Excerpts from Testimony Before the U.S. House of Representatives, 
Committee on Science and Technology, Science Policy Task Force, 
Hearings on “Research Project Selection" 
April 8-10, 1986 


National Institutes of Health 
Excerpts from testimony of Dr. Ruth L. Kirschstein 
Director, National Institute of General Medical Sciences 


Peer review procedures for health research supported by the 
Federal Government were initiated in 1902. At that time, the 
Fifty-seventh Congress established a Scientific Advisory Board of 
non-governmental scientists to assist the Surgeon General in the 
administration of the Hygienic Laboratory. In later years that 
Laboratory was renamed the "National Institutesof Health.” Ara 
the Advisory Committee was reconstituted and renamed the 
"National Advisory Health Council." 


When the National Cancer Institute was established in 1937. 

the Cancer Act also provided the legal basis for the National 
Advisory Cancer Council. The Act stated that the Council was to 
play a Key role in recommending the award of grants. These 
autnorities and procedures were extended to grants and 
Peliowshnhips in health research areas of 311 tne NIH Institutes in 
1944, when the Public Health Service Act was passed. 


In 1946 the director of the NIH recognized the need to establish 
a more formal advisory mechanism for the assessment of technical 
and scientific merit by using a system of initial review groups, 
aise called “study sections." Study section review is the 

first level of the NIH peer review systen. Each study section 
is composed of i4-20 non-governmental ecientists having national 
stature in a particular discipline--for example, biochemistry, 
hematology surgery, genetics, molecular biology. These initial 
reviewers constitute the true scientific peers of the persons 
submitting research grant applications. 


The review groups usually meet three times a year, for 2-3 days 
per meeting. At the meeting, each research grant application is 
presented and discussed individually, in detail, by particular 
assigned reviewers, after which there is general discussion 
leading eventually to the development of a consensus. Each 
application is assessed for such factors as: the objectives or 
the research; the research protocol; the capabilities of the 
investigator and his or her associates: the research environment; 
the probability of success; the appropriateness of the budget to 
the tasks described; and the scientific significance of the 
proposed research. 
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The study section has three options, arrived at by majority vote: 
approval, if the application is deemed to have scientific merit; 
disapproval if it is considerd to be devoid of such merit; or 
deferral. If insufficient information is presented for 
formulation of a fair and rational recommendation, the study 
section may recommend deferral in order to obtain the needed 
information or to resolve any problems. If a recommendation is 
made for approval, then each Study Section member assigns 
privately a numerical priority rating based on his/her individual 
evaluation of the particular application. The possible scores 
range from ito 5, with 1 representing the most favorable 
assessment and 5 the least favorable. The assigned scores are 
then averaged and computed into a final overall “priority score" 
representing the entire group's opinions. 


The initial reviews performed on each research grant application 
are translated into a written “summary statement." This 
includes a critique, a recommendation, and, if favorably 
recommended, a priority score. 


This is the first level of the Review System. However, it must 
be stressed that the NIH peer review system is a dual one. The 
second level occurs subsequent to the first and is carried out by 
the legislatively mandated National Advisory Council. of which 
there is one ror each Inetitute. Each Advisory Council has the 
broad ?unction of advising on the overall programs of the 
Institute as well as the legal responsibility for recommending 
those research projects for which funds are to be awarded. 


A National Advisory Council consists of 12 or more members, some 
or whom are senior scientists and others, community leaders in 
areas relevant to the responsibilities of the particular 
Institute. An Institute may award a research grant only if it is 
recommended for approval by its Advisory Council. If? the Council 
Goes not recommend approval, an Institute may not award the 
research grant. It is the Council's responsibility to review t . 
appropriateness of the assessment by the study section and to 
make firal recommendations regarding program relevance, 
significance, and the priority of the research in relation to t 
Institute's mission. 


By means of this dual system, there is a mandatory separation of 
the assessment of projects purely regarding merit from the 
subsequent policy recommendations as to the scientific areas of 
importance in which projects will be awarded and the level of 
resources to be allocated to those areas. 


2. National Science Foundation 


Excerpts from testimony of Dr. Mary Clutter 
Senior Science Advisor 


Except for a few notable large-scale initiatives such as the 
engineeering research centers and supercomputer centers, most 
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proposals received by the Foundation are not in response to 
specific solicitations or formal requests-for-proposals or the 
type that many other agencies employ. We do, of course, 
publicize our programs extensively, specify the format for 
proposals, and encourage direct contacts with our program staff. 
But the bulk of the proposals are unsolicited: that is, the 
research ideas and methods are those of the applicants. 


The mechanics of proposal handling can be described simply. The 
research programs of the Foundation are organized along the lines 
of fields of science and engineering. In the FY 1987 budget you 
will find some 140 research support programs in 27 Divisions. 

The proposals come into a central unit, where a file is 
established and they are assigned to the appropriate Division for 
handling. They are then further assigned to a program officer. 

It is important to note that these program assignments are made 
by scientists and engineers who Know the subject matter. 


Before I tell you about their important role, let me describe a 
typical prozram officer. Counting Division and Section Chiefs, 
there are about 300, or more then a quarter of the total 
Foundation staff. All of them nave significant research 
experience, and almost all have doctorates. They are recruited 
for the breadth and depth of their specialized Knowledge as well 
as for the personal qualities they will need to deal with the 
prominent scientists in their field and the many other demands of 
the position. 


At any one time, about a third are one- and two-year "rotators" 
on leave from their universities and colleges. The 

Foundation places a very high value on recruiting and using 
“rpotators" because they have recent bench-level experience and 
are able to infuse our decision-making systems with their fresh 
viewpoints. Many or the permanent staff began as “rotators". 


Over the course of several nonths, the program officer oversees 
the external review, evaluates the comments of the reviewers, and 
makes a recommendation as ta whether the proposal should be 
awarded or declined, taking into account other considerations-- 
such as the relationship of a particular proposal to the field as 
a whole and to other pending proposals, and the amount available 
in the program's budget. 


The prcgram officer's recommendations are reviewed at one or more 
higher levels. Approval recommendations are then forwarded to 
specialists in the Division of Grants and Contracts, who check on 
all the nonscientific aspects of the award and issue the formal 
notification to the applicant's institution. In some cases, as 
with a sizable request for funds or, for example, the first group 
of proposals in a new program, the recommendations are further 
reviewed by an Assistant Director and by the National Science 
Board. 


Ensuring the quality, completeness and fairness of the external 
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review is one of the principal responsibilities of the program 
officer. The review process employed depends on the tradition of 
the particular field of research; in the physical sciences and 
engineering, proposals are usually mailed to reviewers who 
respond individually: in the biological, behavioral and social 
sciences, a panel of scientists reviews the proposal in addition 
to the mail review, for large projects and special competitions 
of various types, reviews are often supplemented with site visits 
by NSF staff and teams of outside experts. 


In the mail-out procedure, the program director sends each 
proposal to several people--sometimes as many as ten--whom he or 
she has identified as knowledgeable on the topic. The reviewer 
receives standard instructions and review forms, and responds 
directly to the program officer. Where panels are used, groups 
of experts meet at some specified interval--generally three times 
@ year in the larger programs--to weigh a group of proposals, 
taking into account the comments of the ad hoc reviewers. The 
first criterion in choosing panelists is expertise in the field: 
then we look to balance the panels with regard to such factors as 
institutional size and participation of women and minorities. 
Panelists serve for a specified period of time: commonly, some 
portion of the panel will rotate off each year. 


Both methods orf peer review have pros and cons, and both have 
advocates who strongly defend them and would be very reluctant to 
change. The type of review used in a program has grown out of 
the scientific tradition in that particular field; most 
applicants in a given field are ramiliar with it and understand 
it. On average, no matter which method is used, a proposal has 
the benerit of 5.5 completed external reviews. 


To give you some idea of the scale of operations, during FY 1985, 
the Foundation took final actions on 24,403 competitively 
reviewed proposals. Of these, 7,968 were awarded; almost twice 
as many, 15,504, were declined: and 931 were withdrawn by 
applicants ~rior to a decision. Obviously then, overall, about 
one-third «+ the total completed actions became awards. Some 
programs historically have higher ratios of awards to declines, 
some lower, depending on a lot of factors including the size of 
the program budget and the nature of a typical award in that 
field. 


In Fiscal 1985 a total of 59,725 individual reviewers and 
panelists contributed their services to NSF. They came from a 
wide range of institutions--not only academic institutions, 
large and small, but from industry, Government and even foreign 
research and educational organizations. 


As that total shows, program officers are always on the lookout 
for qualified, Knowledgeable reviewers from all backgrounds. 
There are very practical reasons for that: fresh insights and 
advice can be a valuable contribution to decisions, particularly 
when one is trying to discern the best proposal among many of 
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substantially equal merit: and certainly one does not want to 
overload individual reviewers (however Knowledgeable) with too 
many requests. 


The specific factors that we ask reviewers to take into 
consideration are spelled out for applicants in our publication 
entitled “Grants for Scientific and Engineering Research." 
Basically, the National Science Board has set forth four criteria 
to be applied to all proposals “in a balanced and judicious 
manner.“ They are: 


(1) Research performance competence: That is, the technical 
soundness, the capability of the investigator(s), and the 
adequacy of institutional resources available for the work: 


(2) The intrinsic merit of the research: that is, the extent 
to which the proposed work is expected to lead to new discoveries 
or fundamental advances in its field or across fields of 
research; 


(3) Utility or relevance: the extent to which the work could 
contribute to an extrinsic goal such as a new technology: and 


(4) The effect on the infrastructure of science and 
engineering: that is. what the work will contribute to the 
Nation's research, education and human resource base. 


The relative weights of criteria (2) and (3), intrinsic merit and 
utility. depend on the character of the work. The fourth 
criterion allows the program to take into account such questions 
as the participation of women and minorities, institutional 
distribution, and the stimulation of important but underdeveloped 
fields of research. In many cases programs also use special 
criteria related to their purpose as in, for example, specialized 
programs for equipment grants or visiting professorships for 
women. In any event, the scien*ific merit of the proposed work 
and the competence of the applicunt are the principal factors. 


Reviewers are asked to provide :iot just an overall rating but 
narrative comments, including comments on the researcher's 
earlier completed work where pertinent. These narratives are 
more valuabD. in the ratings themselves, not just to program 
officers in arriving at their recommendations but to applicants, 
who routinely receive them, as I will explain in a moment. 

The program officer s job is only half finished once the reviews 
have been completed. It is easy to make recommendations to fund 
very highly rated proposals and to decline very low rated ones. 
But there is always a middle group, sometimes a sizable one, ofr 
meritorious proposals all of which would be worth funding if 
enough money were available. Because that's rarely the case, we 
call on the program officer to use his or her scientific 
judgement, experience and discretion in selecting among them. 


The Foundation ,-has a tradition of selecting program officers not 
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only for their research experience and other technical 
Qualifications but for their integrity and objectivity. And as I 
indicated, their recommendations are examined in turn by more 
experienced Section and Division Directors familiar with the 
broader scientific and policy environment within which the 
Foundation operates. 


To ensure an open and fair system, we have built in several other 
safeguards, both for the individual applicant and to satisfy 
ourselves that the system as a whole is operating as it should. 


First, Foundation policies and practices about potential 
conflicts of interest, or even the appearance of conflicts. is 
very clear and made Known to reviewers and program officers. We 
ask them to identify any academic affiliations, personal 
relationships or financial interests that could constitute or be 
regarded as a conflict, and to withdraw from the decision process 
if warranted. 


Second, since the mid-1970's we have routinely sent all of 
the reviews to the applicant verbatim but without attribution to 
individual reviewers. This system provides for the interests of 
both applicants and reviewers--reviewers because some of them 
might be reluctant to he entirely candid otherwise: and 
applicants because they find the reviewers' detailed comments 
very valuable for explaining a decision and for reformulating 
their proposal if they wish to resubmit it. 


Third, we have a system for formal reconsideration of 
proposal decisions, at the request of the applicant. This is not 
a “de novo" review of the scientific merits but an examination of 
how the case was handled by the program officer. Reconsiderations 
are carried out by persons not involved in the earlier decision. 
An applicant not satisfied with the initial reconsideration may 
have his or her institution request a further reconsideration by 


the Foundation's Deputy Director. 


We also take measures to examine our process systematically: 
First, staff’ of the Director's office of Audit and Oversight 
routinely examine a small random sample of actions for compliance 
with Foundation procedures and provide direct feedback to the 


program division. 


Second, every one of the programs is given a comprehensive review 
years by a small group of external peers familiar 
with the field. These “visiting committees," if you _ make 
formal reports to the NSF Director about each program. The 

and the annual summary, are publically available. 


every three 


reports, 


These peer oversight committees select their own a. er 
actions and address such matters 4s whether the select ot an 
reviewers, the program officer's rationale for recommen ng : 
award or declination, and the balance among projects selected are 
reasonable. ... The comments of these external peers about the 
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professional stewardship of the N&F atarfr are senerally very 
Pavorable. 


Third, we have undertaken some experimental studies of the 
proposal review and award system as a whole. Our Program 
Evaluation Staff, which is part of the Controller's office and 
thus independent of the Program Divisions, has run various 
statistical tests on carefully-constructed samples to see if the 
system has been "fair" to certain classes of applicants. Here 
are some of the findings: 


(1) For NSF as a whole, about one of every four new 
proposals are funded, and about one of two renewal 
proposals. Generally, renewal proposals receive better 
ratings, but even where the ratings are about the same, the 
renewal is somewhat more likely to be funded. 


(2) Proposals of roughly the same quality as indicated by 
the ratings have the same chance of being awarded or 
declined no matter what states the proposals are from. 


(3) Beyond the ratings, and whether the proposal is 
new or a renewal, none of the following three factors 
have a statistically significant effect on decisions: 
The prestige of the applicant’s department, his or her 
“professional age," or the amount of money requested. 


(4) There are other less global but nevertheless interesting 
results. For example, there is no statistically significant 
difference in the ratings that male reviewers give to 
proposals from women compared to men. 


3. Department of Energy 
Excerpts from testimony of Dr. James S. Kane, 
University of California 
(Former Deputy Director, Office of Energy 
Research, DOE) 


Energy Research is a very large fu: der of research, even by 
federal standards. Its annual research budget exceeds $1.8 
billion, most of it basic research, the remainder being highly 
sophisticated applied science. Energy Research is also the 
principal organization of the federal government in the 
construction and operation of very large facilities used for 
basic research--accelerators, including the ones at Fermilab and 
at the Stanford Linear Accelerator Center, nuclear research 
reactors, synchrotron light sources and other large and 
expensive scientific facilities. So peer review, in Energy 
Research. is used not only to choose the scientists that are to 
be funded, it is used in addition to choose the experiments to 
be done on these expensive facilities, whose operating costs can 
be thousands of dollars an hour. ... The large research 
facilities are used predominantly by university scientists. In 
the case of the high energy particle accelerators, for example, 
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the university community comprises about 75% of the users. 


Before I start describing how Energy Research uses peer review, 
let me first define what I mean by this term. I can identify at 
least five different kinds of peer review, all of 

them used to varying degrees by Energy Research: 


1) Mail reviews of written proposals, done individually by 
several (usually 5 or 6) experts. The proposer does not know 
the identity of the reviewers. These reviews are 
prospective, in that they evaluate promise, not performance, 
although the past record of the proposer is invariably 
considered. 


2) Panel reviews, based on written proposals evaluated by a 
panel of several experts, which collectively rank-orders the 
proposals. These reviews too, are prospective. 


3) Oral panel reviews, in which a panel of experts evaluates 
&@ proposal, usually on the basis of both written and oral 
presentations. These too, are prospective. 


4) Progress evaluations of individual research projects, 
usually by an appointed panel of experts, with a 
presentation by the project scientist. These revicws are 
retrospective. Progress, rather than promise, is evaluated. 


5) Evaluation of group efforts, by a panel of experts. 
Again, the review is retrospective. The evaluation is 
focused on accomplishments and progress. 


As I stated above, Energy Research uses all of these forms of 
peer review. The first three are the most common for research 
sponsored at universities, and the latter two for research at the 
National Laboratories, but there are occasions where process 1) 
is applied to work at the Laboratories, and process 5) is applied 
to work at universities. I think nearly every basic researzh 
project sponsored by th. DOE is subjected to one or more of these 
processes. 


I mentioned earlier that the allocation of time on the expensive 
research facilities at the DOE laboratories is also done by the 
peer review process. Each research facility has a program 
committee of expert scientists. A majority of the committee is 
from institutions other than the one at which the 

facility is located. This committee reviews 311 proposals 

that involve use of the facility, and rank-orders them by merit. 


This procedure ensures that scientists from the site where the 
facility is located do not have an unfair advantage in gaining 
accese to the facility. Most of the ecientists doing experiments 
at a facility are from other institutions. 


4a. Office of Naval Research 
Excerpts from the testimony of Dr. Marvin Moss, 
Director, Office of Naval Research 


Any discussion of ONR‘s project selection process must recognize 
the fact that ONR is a mission oriented sponsor of basic 
research. It encourages the acquisition of fundamental 
Knowledge needed to solve future military problems in areas 

such as communications. surveillance, targeting, propulsion, 
mobility, guidance and control, navigation, energy conversion, 
materials and structures, and personnel support. The Navy of the 
2ist Century is being planned now through the research supported 
by ONR. 


Because of ONR's mission orientation, project selection must be a 
two step process. First, it must establish broad programmatic 
thrusts and priorities reflecting a suitable balance between Navy 
need and relevant scientific opportunity. Next it must select 
specific research projects and tasks to implement those broad 
thrusts and priorities. ... 


Partially because of its unique mission, ONR depends primarily on 
its Scientific Officers for the selection of specific research 
projects. Academic peer reviewers cannot be expected to be 
Knowledgeable about the Naval mission and its research 
implications. It is ONR‘s exceptional cadre of Scientific 
Officers who made its past record of achievement possible. ONR 
Scientific Officers are encouraged, as a matter of policy, to be 
active researchers and to play a leadership role in the 
scientific community while, at the same time, establishing and 
maintaining close communication with the Navy technology 
development and operations communities who will ultimately use 
the products of their research programs. 


The ONR program employs a variety of mechanisms to support 
research at universities. In some cases, spt ial objectives ere 


included in the selection process. For exam’ »°, we recently 
established a program to identify and suppor axceptionally 
promising young investigators. Other speciai programs are 


designed to foster substantive cooperation and interaction 
between research personnel at Naval laboratories and their 
counterparts at universities. However, the vast majority of our 
program is implemented through the mechanism of project contracts 
with individual principal investigators for research projects 
that are selected on the basis of scienti*?ic quality considered 
in the context of Navy need. 


The ONR Scientific Officers play the key role in project 


selection and management. They are given broad discretion in the 
selection of external projects for support, and are then held 
responsible for their results. Although there is no formal peer 


review process at ONR, the Scientific Officers do seek the advice 
of associates within the Navy and of appropriate experts external 
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to the Navy. The methods employed to seek expert advice, which 
may be hizhly structured or informal. are determined by the 
Scientific Officer to meet the particul2r needs of his or her 


program. External reviews, if employed, are advisory. The 
Scientific Officers are ultimately responsible for a project's 
contribution to Navy goals. The decisions of the Scientific 


Officers are reviewed by Division Directors and by appropriate 
science management. However, those decisions are rarely 
reversed. The ONR Scientific Officers are heavily involved in 
project monitoring, including site visits. and it is they who 
must answer for their programs in periodic program review and 
evaluation by upper management. 


The ONR program that has resulted from the processes chat I have 
just outlined has the following characteristics: 


-60 to 65% long-term research to advance the state of 
knowledee sacross the spectrum of disciplines relevant to the 
long-term Navy needs. 


-15% focused on long-term high-risk efforts with high 
potential payoff. 


-20-25% devoted to programs closely associated with specific 
applications with defined transition potential. 


Because of the requirement to select programs that have 
cutstanding scientiv?ic merit and fit into an overall set or 
programmatic priorities, the ONR Scientific Officers cannot be 
Dassive sand ftimpliy react to proposals as received from the 
acsdemic community. They must play a very active role in 
communicating ONR's programmatic interests and priorities to the 
academic community and in seeking out scientific opportunities 
relevant to Navy priorities. Not only do they spend considerable 
time visiting university laboratories and scientists for this 
purpose, they also organize special conferences and workshops and 
monitor and participate in relevant activities of the National 

cademy of Sciences, professional soc‘eties and other 
vozanizations. To do this effectively. they must have 
established a certain level of visibility and stature in their 
research communities. 


On the opposite side of the coin. the Scientific Officers very 
asctively seek out mechanisms to ensure current knowledge of Navy 
and DOD needs. To cite just a few examples, many of our 
Scientific Officers take on responsibilities beyond their 
principal role as Navy research managers to also manage Navy 
development programs. and programs of DARPA and SDIO. In 
addition, they consult on special near term problems faced by 
Naval flicet and operations personnel. 


Before completinse my remarks on the topic of project selection, I 
would like to describe somewhat more structured approaches 
currently employed for two special purpose programs that 
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represent a relatively small but important fraction of our total 
program. I am referring to the ONR Young Investigators Program 
and the ONR URI Block Research Program. Both programs are 
components of DOD's University Research Initiative established 
in FY-86 to strengthen the ability of U.S. universities to 
perform basic research and to train scientists and engineers in 
areas vitally important to the national defense. 


The ONR Young Investigator Program is intended to attract the 
best and brightest young academic researchers to areas of 
research that are important to future Navy requirements by 
providing special recognition and stable funding for research at 
a level consistent with the purchase of equipment and support of 
graduate students. The goal of the program is to provide a 
future resource of informed, committed scientists and engineers, 
and to establish strong long-term ties between DOD and 
outstanding academics. It must be clear that the proposed 
investigator is a potential leading faculty member with a strong 
commitment from the university's officials. Base funding for 
each young investigator is $50,000 per year for three years. As 
an incentive for becoming involved with such Navy research 
activities as the Navy laboratories and the Navy Systems 
Commands, ONR will match on a 2-for-1 basis support gained from 
these sources. ONR matching funds are limited to $80,000, but 
this does not prohibit a young investigator from getting more 
than $40,000 from other Navy sources. 


It is often claimed that young investigators sometimes are placed 
at a disadvantage by "peer" and other conventional project 
selection processes. We wish to insure that particularly 
promising bright candidates receive appropriate recognition and 
encouragement. The program is announced via a specific program 
announcement. One hundred and seventy-nine proposals were 
received in response to our FY-85 announcement. Unfortunately, 
we were only able to fund ten. 


All Young Investigator proposals are initially screened by the 
directors of the ONR divisions responsible for the research area 
identified by the candidate young investigator. Div.sion 
directors in turn present the case for their most promising 
candidates to a selection panel composed of ONR'‘'s Research 
Advisory Board and myself. Young Investigator proposals are 
scored on the basis of creativity of the proposal, past 
performance of the candidate, and evidence of excellence and 
commitment on the part of the candidate's university. 
Consideration is also given to a desire to have a reasonable 
balance of young investigators across Navy relevant areas of 
science and engineering. 


Block Research Programs. As already noted, ONR has traditionally 
emphasized the individual project contract. Broader 
institutional support mechanisms have been employed, particularly 
in the ocean sciences, but even in those cases institutional 
Proposals are evaluated essentially on a project-by-pruject 
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basis. In FY 1986, under our newly established University 
Research Initiative, we will begin to support a number of new 
block research programs at U.S. universities. 


The Navy URI research programs are designed to increase the 
number of science and engineerng graduate students; to increase 
investment in major pieces of research equipment at universities; 
to increase the investment in higher risk basic scientific 
research in support of critical Navy and DOD technologies; and to 
provide more opportunities for contacts between universities, 
industry. and Navy and other DOD laboratories to maximize the 
benefits to be derived from defense research for the nation's 
security. both military and ecoromic. We recently issued a broad 
area announcement, jointly with DARPA, that requests proposals in 
20 broad areas of science and technology critical to the DOD 
mission ranging from mathematics to arctic science and 
engineering. 


The ONR block research program contracts will have a maximum 
life of five years with maximum annual funding of four million 
dollars per year. The block research program selection process 
can be summarized as follows: 


° A proposal evaluation panel will be established for each 
of the research areas identified in the ONR/DARPA broad 
agency announcement. 


° Each proposal evaluation panel will be chaired by an 
appropriate ONR Scientific Officer and/or DARPA Program 
Manager who will ultimately be responsible for 
recommendations and management of the block research 
program(s) awarded in the subject research area. Other 
members of the evaluation panels will include appropriate 
representatives from in-house Naval laboratories, the 
Army, Air Force, DARPA, NSF and industrial research 
organizations. Academics may be included when it is 
clear that conflicts of interest are not present. Where 
‘ppropriate, other interested government agencies may be 
-epresented (e.g. NOAA or NASA in ocean remote sensing). 


° Evaluation panels will review and discuss all proposals 
received in their subject areas and rank them into three 
categories. Site visits (typically 1 1/2 days in 
Guration) will be made for all proposals in the highest 
category. Upon completion of site visits, each of the 
proposal evaluation panels will make recommendations to a 
Program Selection Board consisting of ONR's Research 
Advisory Board supplemented by representatives from 
DARPA, Army, Air Force. DUSR&AT and appropriate Directors 
of Navy Laboratories/R&D Centers and the Director, ONR. 


° Final selection will be made by the Program Selection 
Board. 
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re) All selections will be reviewed by an appropriate DOD 
committee prior to announcement to ensure coordination of 
Army, Air Force, DARPA, and Navy block research programs. 


In order to satisfy the research objectives of DOD and the Navy, 
ONR uses a number of mechanisms to sponsor research. 
Historically, greatest emphasis nas been placed on the individual 
project contract which we still believe represents the best 
mechanism for supporting work of the highest scientific quality 
within the context of naval requirements. Partially because of 
its unique mission, ONR has placed less emphasis on formal peer 
review in its selection process than some other research 
sponsors. However, the Key to the ONR project selection process 
has always been the ONR Scientific (fficer, who is a state-of- 
the-art scientist or engineer, not «x contract administrator. The 
ONR Scientific Officer must be an activist with a broad Knowledge 
of his or her discipline and an understanding of its relevance to 
the naval mission. ONR's record of accomplishment is more the 
direct result of the outstanding quality of its Scientific 
Officers, and the authority given to them, than the result of any 
particular process for project review and selection. 


5. Department of Agriculture 
Excerpts from testimony of Dr. Neville Clarke, 
Director, Agricultural Experiment Station, 
Texas A&M University System 


State Agricultural Experiment Stations receive sizable 
appropriations from the state legislatures, generally in 
accordance with the size and relative importance of agriculture 


in the individual states. In addition, federal appropriations 
are awarded by the USDA in three ways: 1) formula-based funding; 
2) special grants: and 3) competitive grants. Other federal 


funds from non-USDA sources may come from any of the federal 
granting agencies. In addition, the State Agricultural 
Experiment Stations receive varying amounts of industrial funds 
as well as private endowments. To a minor degree, they receive 
some support from selling the excess agricultural products 
associated with their research. 


Funding of agricultural research by the USDA relies on a system 
of peer review for all three kinds of research grants as well as 
explicit methods for accountability and evaluation of research 
productivity. These will be discussed in more detail in 
succeeding parts of this testimony. 


Formula Based Funding. The legal basis for formula funding 
resides in three key pieces of legislation: the Hatch Act which 
was passed in 1887; the Evans-Allen program which was first 


funded in 1967; and the McIntire-Stennis legislation which was 
passed in 1962. Hatch funds are for a broad scope of 
agricultural research; Evans-Allen funds support agricultural 
research in the historically all black institutions; and 
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McIntire-Stennis funde egupport forestry and forest geience 
research. 


Appropriated by Congress, formula-based funds are distributed to 
the individual states by the Cooperative State Research Service 
on the basis of a “formula™ that, generally speaking, has to do 
with the number of farms or enterprises, rural population and the 
size of the agricultural or forest activity in the state. One- 
fourth of the funds appropriated under the Hatch Act must be 
spent on cooperative regional research projects. 


Though the funds are distributed to the individual states on a 
formula basis, it is important to recogrize that individual 
projects are subjected to peer review within the individual 
states. The proposals that are approved at the state level 

must then be approved by the Cooperative State Research Service 
before formula funds can be spent on the projects. In the case 
of regional research projects, a national committee, called the 
Committee of Nine, reviews and approves regional projects as well 
as interregional projects that utilize Hatch Act funds. 


Individual research projects are evaluated by annual progress 
reports submitted by the scientists and by periodic reviews. In 
addition, the Cooperative State Research Service helps the 
individual State Agricultural Experiment Stations organize and 
conduct in-depth reviews of a total program or subdject matter 
area approximately once every five years. 


Perhaps one of the most important aspects of formula-based 
funding is the intentional commingling of state and federal funds 
to support agricultural research. This is a fiscal manifestation 
of the unique state-federal partnership that has provided the new 
Knowledge necessary to make U.S. agriculture the most productive 


in the world. This partnership allows a very substantial 
leveraging of federal funds that are provided for research within 
the states. The criteria are well established for approving the 


use of federal formula funds for research that applies to a 
multi-state ob.) ctive or to a national problem. 


The individual research programs that are partially sponsored by 
formula-based federal funding are also dynamic in terms of the 
turnover of research projects, which occurs every three to five 
years. Although some funding stability is necessary for long- 
term agricultural research such as the development of new plant 
and animal germplasm, there is, within the formula-based funding 
activity, a substantial flexibility of resources, allowing 
adjustments with redirection of efforts toward critical issues. 
When these areas of research are viewed in the aggregate from a 
national perspective, it is sometimes difficult to portray the 
mobility that actually exists in individual states or in 
individual projects. 


One of the major characteristics of formula-based funding is the 
utility of these funds in early problem-solving and in developing 
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awareness of emerging agricultural problems and opportunities 
that need to be addressed by research. There are numerous 
examples of the timely initiation of new research with formula- 
based funds; research which has provided the infrastructure and, 
in some cases, the pilot studies that have identified the need 
for new initiatives subsequently funded by either special or 
competitive grants. Changes in the base program of research 
funded by formula-based resources come not only from internally 
initiated perceptions of need, but may also be externally driven 
by interactions with the commodity groups, the Congress and the 
Administration. There are a number of examples that could be 
cited to illustrate the dynamics of the base program. ... 


Special Grants.. The USDA's Cooperative State Research Service 
has awarded special grants over the years for research in areas 
of need or opportunity targeted by the Congress. Over the years, 
many of these specisl grants were highly specific to research 
needs and problems in individual states. Increasingly, the 
special grants are funding projects more regional and national in 
scope and application. Special grants are also now awarded on 
the basis of a peer review process that ensures the scientific 
merit of proposals submitted for funding. Special grants, 
however, are a relatively small part of the total USDA budget for 
cooperative research with the states. In FY86, $28.6 million was 
provided to USDA by the Congress for special grants. 


Competitive Grants. The 1977 Farm Bill initiated the USDA's 
competitive grants program and the funding for it has continued 


to increase in the intervening years. The USDA modeled its 
program after the National Science Foundation's avproach to grant 
evaluation and award. The State Agricultural Experiment Stations 


enthusiastically supported this program once initial concern was 
allayed that funding for the competitive grants program might be 
transferred from formula-based funding for the base programs of 
research and to provide new recources for the competitive grants 
program. 


In general, competitive grants have been made available for 
fundamental biological research that is critical to agriculture. 
Because of the basic nature of this work, the USDA's policy 
allows competition from both within and outside the traditional 
land-grant institutions. Scientific peers are actively involved 
in all phases of management, including helping to set the 
research agenda, evaluating proposals, awarding grants and 
evaluating progress. 


The competitive grants program has also been quite dynamic. An 
esrly emphasis in plant sciences has broadened to include other 
areas of biological science. A major initiative in the area of 
biotechnology was proposed by the Administration and first funded 
by the Congress in FY85. The biotechnology initiative is still 
emerging with strong support by the White House Office of Science 
and Technology Policy and others in the Administration. The 
competitive grants program has undergone substantial growth; more 
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than $55 million will be awarded in FyY86. Thus, almost 20 
percent ofthe USDA funds distributed to the states through the 
Cooperative State Research Service are now being awarded in the 
competitive grants program. These grants complement funding 
provided by other federal reseerch agencies such as NSF, HHS, EPA 
and the Department of Interior, for broad, fundamental 
agricultural research. 


APPENDIX II 


NSF's Program Oversight Process 
Report for FY 1985 


_ = 


~ nsf Februarv 25, 1986 NSB 86-45 


OFFICE OF THE 
DIRECTOR 


MEMORANDUM TO MEMBERS OF THE NATIONAL SCIENCE BOARD 


SUBJECT: Annual Report on the Foundation's 
Use of Peer Review 


This report is submitted in accordance with the policy Regarding 
Peer Review Endorsed by the NSB at its 188th Meeting on March 17- 
18, 1977 and Amended at its 25lst Meeting on March 15-16, 1984 
which states: 


VI. The Director shall provide the Board no less 
than annually a report on the Foundation's 
use of peer review. This report shall 
include: 


A. A list of the standard classes of 
exemption from peer review and 
information on the waiver of peer 
review in other cases. 


B. Summary of external peer oversight 
committee reports. 


C. Recommendations for change or 
further consideration of the 
Foundation's policies on peer 
review. 


D. Such other information as the 
Director may feel appropr-ate. 


It is the policy of the National Science Foundation to 
make available, upon request, a list by division of all 
reviewers used during the preceding year. 


The list of the standard classes of exemption from peer review is 
prescribed in NSF Manual 198 given in Attachment I. Although 
recast in a new format, the exemptions are the same as those 
given in the report presented last year except for the addition 
of the words “or under the auspices of international cooperative 
scientific programs" to exemption 3(b) covering proposals for 
certain conferences, symposia, ard workshops. This change was 
made to facilitate planning conferences in international coopera- 
tive programs. 
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In FY 1985, there were three cases of special waiver of peer 
review. One was for the support of the NAS-NRC Numerical Data 
Advisory Board. Proposals for support of NAS-NRC Committees are 
usually reviewed by the appropriate NSF Advisory Committee, but 
in this case so many different disciplines are involved that such 
review would be impractical. The other two were Earth Sciences 
planning proposals for multi-institutional consortia to manage 
large projects. These were Incorporated Research Institutions 
for Seismology (IRIS) and Deep Observation and Sampling of the 
Earth's Continental Crust (DOESECC). Since each consortia 
involves essentially all of the institutions involved in these 
areas, reviewers would have conflicts of interests. Both 
projects have been discussed extensively by the Advisory Com- 
mittee for Earth Sciences and the NSB Committee on Programs and 
Plans. 


With my approval at the end of 1985, the Engineering Directorate 
announced that it would provide a limited amount of support for 
proposals of $308,800 or less that involve exploratory research 
without external peer review. This is an experimental approach, 
limited to no more than 5% of any program's FY 1986 budget. The 
objective is to provide a rapid response to proposals to explore 
innovative ideas. After about a year, this special procedure 
will be evaluated. The instructions issued by the Engineering 
Directorate are contained in Attachment II. 


The annual summary of external peer oversight reports is con- 
tained in Attachment III. 


There are no recommendations for changes in policies on peer 
review at this time. 


= 
Erich Blo 
Director 


Attachments 
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ATTACHMENT I, app. If 


tional gubdelincs must be applicd. 

a. Criterion of Need. Evidence should be provided, if a 
new administrative structure is proposed, that it is needed to 
address scientific problems in y manner or on a scale not 
possible with existing structures 

b. Criterion of Long-Range Potential. Evidence should be 
provided of a mission with such potential for high scicntific 
productivity over an extended time period that a significant 
number of excelicnt scientists are willing to commit their careers 
to i. 


122 Use of Peer Review 


122.1 General Policy 

The evaluations of formal proposals for NSF funding must 
include external (to the Foundation) peer review, with the fol- 
lowing exceptions: 

a. proposals submitted in response to those formal solic- 
itations that are governed by the Federal Acquisition 
Regulations. 

b. proposals to provide goods or services normally ob- 
tained through procurement mechanisms such as contracts, 
purchase orders, or requisitions. 

¢. proposals for which peer review has been waived by the 
Director or a designee. The responsible Assistant Director may, 
however, require peer review in particular cases. Situations for 
which peer review has been waived include: 

1. the following cases which have been already effec- 
lively peer reviewed: 

(a) incremental funding amendments (such as con- 
tinuing grants) to awards within the time period and scope for 
which a prior commitment was made on the basis of external 
peer review. 

(b) no cost extension of projects. 

(c) supplements, as defined in PAM 404.5, including 
proposals submitted under international Cooperative research 
programs for which support is requested to add an international 
dimension to an active domestic research project presently 
funded by NSF without significantly changing the nature and 
scope of the rescarch. 

(d) awards to cover increased costs duc to changes in 
NSF grant administration requirements or indirect cost 
adjustments. 

(c) awards for which funds are transferred to NSF by 
another agency to augment specific NSF awards. 

(1) awards occasioned by a Principal Investigator’s 
change of institution, provided there is no substantial increase in 
level of funding or change in project objectives. 

(g) ship operations in support of peer revicwed 
projects. 

(h) services to a scientific community, the nature, 
quality. scope and cost of which have been cxamined within 
three years prior to the award by the appropriate advisory 


committee, (A copy of the pertinent minutes will be included 
instead of reviews.) 


2. The following cases where peer review is not 
applicable: 

(a) proposals for Intergovernmental Personnel Act 
(IGPA) awards. 

(b) services provided by another Federal agency in 
support of Polar Programs. 

(c) modifications, improvements, or additions to 
ships in the academic fleet when required for reasons of safety. 

(d) contracts and interagency agreements for surveys 
and data processing in support of the science resources statistics 
systems, for international scientific exchange programs and 
multinational scientific organizations, and for evaluation of fel- 
lowship candidates. 

(e) proposals that (1) are not within the areas set forth 
for support in, or otherwise do not comply with the requirements 
of, the program announcement or solicitation under which they 
were submitted, and are not appropriate for transfer to other 
Foundation Programs: or (ii) are explicitly excluded by NSF 
policy as set down by the National Science Board or the Direc- 
tor, NSF (e.g., clinical medical research, foreign institutions not 
meeting the criteria of PAM 330 “Support of Research In. or In 
Cooperation With, Foreign Countries”). These proposals are 
returned to the proposer rather than being declined without peer 
review. 

3. the following cases where peer review ts impractica- 
ble or too costly: 

(a) international travel grants either to an individual 
oF an Organization in support of a group. 

(b) awards for conferences, symposia, and work- 
shops where reviews cannot be obtained from persons who are 
both knowledgeable and uninvolved or which are undertaken for 
planning and assessment or under the auspices of international 
cooperative scientific programs. 

(c) Logistic Support Only awards when no significant 
additional expenses are involved. 

(d) proposals of less than $20,000 designed to cap- 
italize on unexpected and short-lived natural phenomena when 
the responsible Assistant Director has determined that external 
peer review would result in a delay which would make the 
research unfeasible. (A copy of such determination ts to be 
placed in the file instead of a review and copies of the determina- 
tion are to be sent to the Director and the Division of Audit and 
Oversight.) 

d. proposals which present an extraordinary problem in 
obtaining external peer reviews. (In these few cases the deter- 
mination whether external review is required is made by the 
Director or designee on a case-by-case basis. A copy of the 
determination is placed in the file instead of a review.) 

¢. proposals which are withdrawn before decision. 
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NATIONAL SCIENCE FOUNDATION ATTACHMENT II; 


WASHINGTON. DC 20550 app. Il 
OFFICE OF THE 
“FOR ENGINEERING. AD/ENG BULLETIN NO. 85-25 
GRANTS AND CONTRACTS November 20, 1985 
SUBJECT: Expedited Awards for Novel Research 


One of the goals of the Engineering Directorate is to provide an environment 
which will stimulate more creative and innovative engineering research. As 
part of this effort, the Directorate will provide a limited amount of support 
for proposals of $30,000 or less designed to capitaiize on opportunities for 
exploratory research at academic institutions. Such research includes work on 
the initial elements of a subject in an emerging area of science or 
engineering, exploration of new ideas for cross-disciplinary approaches to 
research by investigators from different fields of science/engineering, or 
exploration of new methodologies for conducting basic research on current 
scientific and engineering problems. Proposals will be evaluated in a timely 
manner so that work can commence as soon as possible. The grants will have a 
duration of one year. 


The Director has authorized the Engineering Directorate to conduct an 
experiment in making such awards without external peer review. The amount of 
funds that can be expended this way in any program is limited to 5% of the 
program's Fiscal Year 1986 current operating plan. This is an upper limit; 
the actual number of awards made in any particular program will depend on the 


quality of proposals received. These funds are not transferable among 
programs or divisions. 


An announcement will be made to publicize this opportunity and explain 
procedures for submitting proposals to the research community. Proposals will 
have to meet the conditions set out in Grants for Scientific anJ Engineering 
Research (NSF 83-57) including a cover page, statement of other research 
support, and budget. The project description should be brief (0 nore than 
two pages) and include a discussion of the importance of the wors, its 
exploratory nature, and why it is important that work proceed as soon as 
possible. ly 2 copies of the proposal are required. Investigators are 
encouraged to discuss their ideas with the appropriate Program Director either 
by letter or telephone prior to submitting a proposal. 


If a Program Director receives a regular research proposal which might benefit 
from exploratory work, the Program Director can suggest that the proposer 
withdraw the existing proposal and resubmit an exploratory research proposal 
which would meet the guidelines of this initiative. Grants funded under these 
provisions are not renewable. However, a new proposal based on the 
exploratory work done under this grant may be submitted following the regular 
proposal submission procedures which require peer review. 
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All proposals for exploratory research will be logged into the MIS upon 
receipt. A special program reference code (9237 ) should be used to identify 
proposals as part of this special initiative. Procedures for processing 
exploratory research proposals will parallel those used for 
conferences/workshops that do not have review. A Form 7 shall be used and it 
shall contain a notation that this proposal is not subject to external peer 
review in accordance with Engineering Bulletin 85-25, In addition, there 
should be a discussion of the merits of the proposal which justify the 
proposed action (award or decline), and an explanation of why the Program 
Director believes the proposal meets the criteria for exploratory research. 
Proposals should be evaluated on the basis of the idea presented rather than 
the Program Director's personal knowledge of the P.I. Division Directors are 
expected to monitor this carefully. Any guidance concerning a particular 
proposal given to a Program Director prior to his/her recommendation for award 
or declination shall be in writing and included in the proposal jacket. It is 
expected that these proposals will be evaluated by the Program Officer within 
one month of receipt by NSF. 


All jackets must be signed by the Division Director to indicate cognizance of 
the proposed action, to confirm that the topic of the award is appropriate for 
the program involved, and that correct procedures have been followed. The 
Division Director is expected to act on the Program Director's recommendation 
within 2 days. A copy of the Program Director's justification for 
recommending an award and the extra copy of the proposal should be forwarded 
to AD/ENG when the jacket is sent to the Division of Grants and Contracts. 


Program Directors are encouraged to use this authority in appropriate 
circumstances to enable a quick response. It is expected that proposers who 
are to get an award will be notified by telephone by the Program Director. 
In institutions which have an Organizational Prior Approval System (OPAS) and 
which are prepared to exercise the 90 day pre-award provision, the principal 
investigator will be able to start as soon as the institution authorizes the 
90 day pre-award process. 


The experimental use of this procedure will be evaluated after it has operated 
for a period of time to decide whether or not it is effective in fulfilling 
the goal of establishing a more innovative and creative engineering research 
culture. Grantees will be requested to provide a brief final report which 
will summarize significant findings resulting from the exploratory work, and 
the principal investigator's assessment of what further work can be done on 
the research topic and the impact this work will have on its particular field 
of engineering. These reports will be used for publicity and to evaluate the 
merit of the program. Program Directors are expected to identify significant 
research results from these awards which can be used for publicity and forward 
these news items to Mrs. Mary Poats. 


This procedure will be effective through Fiscal Yeaft 1986. 


Nam P. Suh 
Assistant Director 
for Engineering 


Distribution: 
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NATIONAL SCIENCE FOUNDATION ATTACHMENT Iil 
WASHINGTON.DC £20550 app. II 
Division of Audit and Oversight 


MEMORANDUM 


DATE: February 19, 1986 


REPLY TO 
ATTN OF: Director, DAO 


SUBJECT: FY 1985 External Peer Oversight Reviews 


VIA: Controller Hk 


TO: Director 


Attached is the summary report on External Peer Oversight for FY 
1985 prepared by the Division of Audit and Oversight. 
[Y Firge 
Jérome H. Fregeau 
Attachment 


cc: Executive Council 


NATIONAL SCIENCE FOUNDATION 
WASHINGTON DC 20550 


Division of Audit and Oversight 


MEMORANDUM 


DATE: February 19, 1986 


REPLY TO 
ATTN OF: Head, Oversight Section, DAO 


SUBJECT: FY 1985 Summary of External Peer Oversight Reviews 


TO: Director, DAO YA 
/ 


The attached report fulfills the DAO responsibility, assigned in 
NSF Manual No. 1, ADM-VIII-300 (formerly NSF Circular No. 147), 
to prepare the annual summary of external peer oversight review 
for the Director to transmit to the NSB. 


During FY 1985 external peer oversight committees reviewed 36 NSF 
programs and provided a positive evaluation of the technical 
stewardship for each. 0 : 
e ? 
lyre 
mm J 
James J. Zwolenik 


Attachment 


February 18, 1986 


SUMMARY 
EXTERNAL PEER OVERSIGHT REVIEWS 
FY 1985 


Abstract 


External peer oversight committees reviewed 
the technical management of 36 NSF programs 
and rendered a written, positive evaluation 
for each program. 


Introduction 


NSF Circular No. 147 on “External Peer Oversight" requires a 
triennial program review by an external peer oversight committee 
for each program in the Foundation.1 The purpose of these 
reviews is to provide NSF management with an advisory appraisal 
of the technical stewardship in each program. External peer 
Oversight committees transmit their reports to the Director 
throughout the fiscal year. Annually the Division of Audit and 
Oversight (DAO) prepares a summary of these external peer 
oversight reports for the Director to transmit to the National 
Science Board. 


Fy 1985 External Peer Oversight Reports 


Thirty-six reports were received covering FY 1985 external peer 
Oversight review findings. All reviewed programs appear in the 
left-hand column of Table 1 (attached) and are distributed among 
directorates as follows: 


AAEO 5 MPS 1@ 
BBS 15 SEE g 
ENG 4 STIA 2 


Fi.ty-one reviews were scheduled by the directorates for FY 1985. 
Thirty-six were submitted giving a submission rate of 70%. Of 
the 15 scheduled but not submitted, 7 were not required until FY 
1986 or later and represent no problem. Of the remaining 8, 3 
programs in the Division of Information Science and Technolosy, 
BBS, were overdue in FY 1985 and 5 STIA programs (3 in the 


1 On July 12, 1985 NSF Manual No. 1, ADM VIII, Subchapter 
300, replaced NSF Circular No. 147. The substance of 
the requirements remains the same. 
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Division of Policy Research and Analysis and 2 in the Division of 
Science Resources Studies) were due in FY 1985. DAO has no 
record of approval for these 8 postponements. All eight 
delinquent programs have scheduled oversight reviews for FY 1986. 


Completeness of the FY 1985 Reports 


All reviews emphasize the technical (disciplinary) aspects of 
program management. 


The review elements prescribed in NSF Circular No. 147 are column 
headings in Table 1, which is a check for completeness. Two of 
the 36 reports (6%) made no answer to one or more of the required 
questions in Element 3. This is only half the number of reports 
which omitted one or more questions in FY 1984. In general both 
the completeness and the quality of responses to required 
questions have improved noticeably over the last several years. 


Generalizations and Notable Comments in the 
Peer Oversight Reports 


The material presented below is divided into the constitutive 
eiements required by NSF Circular No. 147 for each peer oversight 
report. 


ELEMENT 1 - A BRIEF DISCUSSION OF THE WAY IN WHICH THE REVIEW WAS 
CONDUCTED. 


Responses to this question indicate a variety of committee 
Organizations and operations. Many committees held in-depth 
Program briefings prior to studying jackets; others did not; and 
there was a distribution between these extremes. Program review 
committee size ranged between 2 (8 instances) and 1l members (1 
instance). The median-size committee had 4 members. Most 
Oversight reviews were performed by Foundation advisory commit- 
tees or subcommittees augmented by a few additional experts to 
ensure breadth of review. 


ELEMENT 2 - THE NUMBER OF JACKETS REVIEWED AND THE WAY IN WHICH 
THEY WERE SELECTED. 


The number of jackets reviewed for a single research program 
ranged from 7 to 148. At least 1,260 jackets (lower limit) were 
reviewed. 


No uniform procedure for sample selection operates across 
programs. The review committees designed their samples to 
address those points judged to be important in the programs. 
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ELEMENT 3 - THE ANSWERS TO THE SPECIFIC QUESTIONS ASKED IN 3.e. 
OF NSF CIRCULAR NO. 147-- 


(i) IS THE REVIEW PROCESS FUNCTIONING PROPERLY IN TERMS OF 
THE SELECTION OF AN APPROPRIATE NUMBER OF TECHNICALLY QUALIFIED 
REVIEWERS AND ADEQUATE QUALITY AND FREEDOM FROM BIAS OF THE 
REVIEWS? (NOT APPLICABLE TO NATIONAL CENTERS, LOGISTICS SUPPORT, 
ETC.) 


All committees responded to this question. Their 
replies in general were positive. 


One committee reported that review of “group proposals" did 
not follow NSF requirements. (DAO determined that documentation 
On group proposals may not have been as clear as possible in this 
program. The cognizant Division Director has introduced new 
Gocumentation which shows explicitly how NSF requirements are 


being followed.) 


Interestingly, about half of the committees also made one or 
more of the following points along with their positive responses: 


- Low ad hoc reviewer response rate. 
- Insubstantial character of some ad hoc reviews. 
- Panel summaries not substantial, uninformative. 


Although these three observations concern actions of the relevant 
research community, review committees also alerted NSF to insist 
on adequate numbers of “quality” reviews. 


(ii) (a) ARE THE DECISIONS OF THE PROGRAM OFFICER UNDER- 
STANDABLE FROM THE DOCUMENTATION ACCOMPANYING A RECOMMENDATION? 


(b) ARE THE DECISIONS APPROPRIATE AS TO AWARD OR 
DECLINATION AND AS TO SCOPE, DURATION, AND SIZE OF THE PROJECT? 


(Cc) DID PROGRAM OFFICERS TAKE INTO ACCOUNT THE PI'S 
PERFORMANCE ON PRECEDING AWARDS? (DOCUMENTATION SHOULD BE IN THE 
GRANT FOLDER, SPECIFICALLY ON NSF FORM 7, “REVIEW RECORD.") 


All committees responded to parts (a) and (b) of 
this question. For part (a) regarding documentation, the 
committees found the standard of documentation in program jackets 
ranged from acceptable to well-documented. Within this accept- 
able range, one committee cited some instances of document 


Omission and Sloppiness; another committee asked for fuller 

documentation to substantiate decisions on “border line" pro- 

ong — another noted an unevenness in the review analysis on 
orm 7. 
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For part (b) of this question on whether the 
program decisions were appropriate, no committee disagreed 
outright with any award or declination. However, one committee 
in reviewing two programs indicated that it "questioned" 2-3 
decisions. This is the closest any committee came to disagreeing 
with a decision, These 2-3 represent @.2-9.3% of the decisions 
reviewed in FY 1985, One committee found that the budget range 
On awards was rather narrow; and, another that some decisions on 


reduced budgets and on continuing versus year-by-year grants were 
not entirely clear. 


. Only 21 reviews addressed part (c) by commenting 
negatively or positively or by noting the application of this 
requirement for the future. This 60% response rate and the 
content of the response reflect the fact that the NSF official 
requirement for documentation on previously supported research 
was issued May 29, 1984--8 months into FY 1984 which was the base 
year for most of the external peer oversight reviews completed in 


FY 1985. 


(i111) ARE ANY SIGNIFICANT IMBALANCES SEEN IN THE PROGRAM(S) 
UNDER REVIEW SUCH AS SIZE VERSUS NUMBER OF AWARDS, SUBJECT MATTER 
DISTRIBUTION OF AWARDS, AGE, OR GEOGRAPHIC DISTRIBUTION OF 


PRINCIPAL INVESTIGATCRS/PROJECT DIRECTORS? 


The response rate here was 108%. Most found an 
“appropriate balance” or reported "no significant imbalances 
Overall"--frequently adding “under current budgets." Some took 


this opportunity to identify new research areas which fall within 
the scope of the program and which deserve new and/or increased 
emphasis. Committees reviewing 5 programs in BBS found a “non- 
risk" bias in much of the funding. Six reviews advised that 
young investigators should not be penalized for their lack of 
experience; since there was no suggestion of such a bias, this 
remark is a cautionary warning. 


(iv) WITHIN THE LIMITS OF ITS BUDGET, IS (ARE) THE PRO- 
GRAM(S) MEETING THE OBJECTIVES SET BY NSF? (IF POSSIBLE, 
OVERSIGHT REVIEW SHOULD COMMENT ON THE RESULTS COMING FROM THE 
PROJECTS SUPPORTED BY THE PROGRAM.) 


Almost all replied "yes, within limitations of current 
budgets." Committee replies were frequently based upon indivi- 
dual judgments of the impact of NSF support on their fields-- 
many times relying on journal publications. However, sometimes 
the basis of the committee judgment on impact wasn't explicitly 
stated. Some reported insufficient time and/or data to allow 


full or adequate consideration of the significance of results 
from projects sponsored by the program. Two program reviews 
failed to respond to this question; five made a partial response. 
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(v) ARE THERE ANY ADDITIONAL QUESTIONS WHICH SHOULD BE 
ADDRESSED IN AN OVERSIGHT REVIEW? (ASSISTANT DIRECTORS MAY WISH 
TO ASK ADDITIONAL QUESTIONS SUCH AS ONES RELATING TO THE BALANCE 
BETWEEN THE PROGRAM(S) UNDER REVIEW AND OTHER PROGRAMS IN NSF OR 
OTHER AGENCIES AND TO CHANGES TAKING PLACE IN A FIELD WHICH WILL 
AFFECT FUTURE PROGRAM BALANCE.) 


In some instances, additional questions were posed to 
review committees and were answered within a program-specific 
context. In other instances, the review committee formulated its 
Own questions and included replies here. Only 36% of the reports 
addressed additional questions. 


ELEMENT 4 = AN OVERALL EVALUATION OF THE STEWARDSHIP OF THE 
FOUNDATION STAFF IN MANAGING THE PROGRAM. 


Positive evaluation statements were made on the technical 
management of all programs under review. There were no indica- 
tions of unsatisfactory technical management although there was 
one rather negative case. Overall, there was strong endorsement 
of the performance of the technical staff. 


ELEMENT 5 =- ANY OTHER POINTS THE OVERSIGHT COMMITTEE WISHES TO 
ADDRESS. 


About 80% of the reviews mentioned matters of particular 
concern to the oversight committee. Most frequently these were 
elaborations or developments of points triggered by earlier 
questions. The topics were very diverse. Im contrast to FY 1984 
when the program reviews were principally concerned about more 
program funds, FY 1985 showed a much broader range of concerns. 


DAO Conclusions/Recommendations 


Based on the 36 External Peer Oversight Reviews received in FY 
1985, DAO makes the following conclusions and recommendations: 


1. NO MAJOR PROBLEMS WITH PEER REVIEW AND THE NSF DECISION- 
MAKING PROCESSES WERE IDENTIFIED IN THE FY 1985 EXTERNAL PEER 
OVERSIGHT REPORTS. 


2. Oversight review reports have improved tremendously in 
presentation and usefulness over the last seven fiscal years. 
However, DAO feels that closer adherence to the report format 
specified in NSF Manual No. 1 would facilitate more complete 
communication between the oversight committee and USF management. 


RECOMMENDATION: External peer oversight review 
committees should be strongly encouraged to observe the 
report requirements as set out in NSF Manual No. l, 
ADM-VIII-36@ and to format their reports according to 


these requirements. 
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3. Comparatively few review committees have received a set of 
questions related eithes to priority setting within or among 
Programs or to the identification of changes which will affect 
the future balance in a program or among a set of programs. 


RECOMMENDATION: In view of management needs to 
establish priorities and to plan, Assistant Directors 
Should consider more effective deployment of specific 


questions. 


James J. Zwolenik 
Head, Oversight Section 
Division of Audit and Oversight 


Attachments - Table l 
NSF Circular No. 147 
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TABLE 1 
Check of Contents, FY 1985 External Peer Oversight Reports: 
Requirements of NSF Circular No. 147, Sec. 3 (Policy), Part f (Reports) 


Element 1 Element 2 Element 3 Element 4 
r ™ 
Reported # S p E C I F I c Provided 
1Division/ Discussed Jackets Q U E S T I 0 N S Overall 
|Program How Review! Reviewed and Evaluation 
Conducted How Selected (i) (ii) (iiid (iv) (v)1 |of Stewardship 
| 1 T 
AREO a bes 
ATM 
- Atmospheric x 7 x x x O x x x x 
Chemistry 
- Climate Dynamics a x x x x O x x 0 xX 
> Experimental x 7 x x x O x x x x 
i. Metecrology 
o 
DPP 
- Polar Atmospheric 7 x x x x xX x x x x 
Sciences 
- Polar Biolegy and x a x x x xX x a x x 
Medicine 


1 NSF Circular No. 147 imposes no requirement on NSF management to ask additional questions nor 
for the review committee to pose any. 


© NSF Circular No. 129 (Rev. 3) required program officers on May 29, 1984 to document their own 
evaluation of the quality of the investigators previous research accomplishments. This is well ante 
FY 1984, the year most external oversight committees reviewed. 
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Element 1 Element 2 Element 2 Elemerit 4 
Reported # S p E C I F I Cc Provided 
Divisicon/ Discussed Jackets 0 U E S T I 0 N S Overall 
Program How Review! Reviewed and Evaluation 
Conducted How Selected (1) (ia) (iii) (iv) (v)1 lof Stewardsrin 
FES abeé 
ESR 
- Systematic Biology x x x x x Xx a x 0 x 
~ Bisclegical Research x . x x x Xx x 7 0 x 
Resources 
BNS 
- Anthropology 7 x x x x QO x x x x 
SES 
- Law & Social Science 7 / x x x xX x x x x 
- Geography & Regional x x x x x O x x 0 x 
Science 
1 

1 NSF Circular No. 147 imposes no requirement on NSF management to ask additional questions now 
for the review committee to pose any. 

& NSF Circular No. 129 (Rev. 3) required program officers on May 29, 1984 to document their own 


evaluation of the quality of the investigators previous research accomplishments. This is well ints 
FY 1984, the year most external oversight committees reviewed. 
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Element i Element 2 Element 3 Element 4 


Reported # S p E Cc I F I Cc Provided 
Division/ Discussed Jackets Q U E Ss T I 0 . Ss Overall 
Program How Review! Reviewed and Evaluation 
Conducted How Selected (i) (iid (iii) (iv) (v)1 lof Stewardship 


- Measurement Methods x x x x x oO x x 0 x 
& Data Improvement 


DCE 

- Eukaryotic Genetics x x x ix x xX x x x x 

- Cell Biology x x x x x xX x / x x 

- Cellular Physiology | x x x x x xX x ) x x 

- Developmental x x x x x xX x / x x 
Biology 

- Regulatory Biology x 7 x x x xX x 0 x x 


1 NSF Circular No. 147 imposes no requirement on NSF management to ask additional questions new 
for the review committee to pose any. 
© NSF Circular No. 129 (Rev. 3) required program officers on May 29, 1984 to document their own 
evaluation of the quality of the investigators previous research accomplishments. This is well ants 
FY 1984, the year most external oversight conmittees reviewed. 
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Element 1 Element 2 Element 3 Element 4 
Reported # S ~ E Cc I Fe I Cc Provided 
Division/ Discussed Jackets 0 U E Ss T I 0 N Ss Overall 
Program How Review! Reviewed and Evaluation 
Conducted | How Selected (i) (ia) (iii) (iv? (v)1 Jof Stewardship 
| —_ if T 
a bce 
DMB 
- Biochemistry a 7 x x x O x / 0 x 
- Biophysics x 2 x x x O x x 0 x 
- Metabolic Biology x x x x x O x / 0 x 
- Prokaryotic Genetics x x x x x oO x 7 0 x 
. 
oF) 
w 


1 NSF Circular No. 147 imposes no requirement on NSF management to ask additional questions ror 
for the review committee to pose any. 


© NSF Circular No. 129 (Rev. 3) required program officers on May 29, 1984 to document their own 
evaluation of the quality of the investigators previous research accomplishments. This is well into 
“Y 1984, the year most external oversight committees reviewed. 
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Element 1 Element é2 Element 3 Elemert 4 
j Reported # Ss e E Cc 1 F I Cc Provided 
Diavision/ Discussed Jackets Q U E Ss T 1 0 N Ss Overall 
Program How Review! Reviewed and Evaluation 
Conducted | How Selected (i) (ia) (iii) (iv) (v)1 lof Stewardship 
| 1 
abcé 
ENG . — 
CBTE 
~ Kinetics & Catalysis 7 x ¥ x Xx 0 2 x 0 x 
- Multiphase & Inter- x x x x Xx 0 a x 0 x 
facial Phenomena 
- Thermodynamics & x x x x Xx 0 x x 0 x 
Transport Phenomena 
a 
2 - Thermal Systems & 7 x x x Xx . x x 0 x 
Engineering 


1 NSF Circular No. 147 imposes no requirement on NSF management to ask additional questions nor 
for the review committee to pose any. 


€ NSF Circular No. 129 (Rev. 3) required program officers on May 29, 1984 to document their own 
evaluation of the quality of the investigators previous research accomplishments. This is well ints 
FY 1984, the ywar most external oversight committees reviewed. 
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1 NSF Circular No. 
for the review committee to pose any. 


© NSF Circular 


evaluation of the quality of the 
Fy 1984, the year most external oversight committees 


No. 


147 imposes no requirement on NSF management to ask additional 


129 ‘Rev. 3) 


required program officers on May 29, 


reviewed. 


E ler Element 2 Element 3 Elemerit 4 
| Reported # oI E I F I Provided 
Divisiocn/ Discussed Jackets 0 U E S I 0 N S Overall 
Program How Review! Reviewed and Evaluation 
Conducted | How Selected (i) (ii) (iii) (iv) (v)l Jof Stewardship 
7 | 
MPS . tb > T T 
: 
DCR 
- Computer Systems x A 2 x Xx x x x 
Design 
- Intelligent Systems x x xX x Xx xX x x 
- Software Engineering x + K x Xx / / x 
-~ Computer Research x x x x x x x x 
Equipment 
~ Software Systems xX x x x Xx / x x 
Science 
- Special Projects x x x x x x x y 
1 en 


Questions 1 


1984 to documerit their own 


investigators previous research accomplishments. This 


1s 


well ant: 
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Element i Element < Element 3 Element 4 
Reported # S we Ee Cc I F I Cc Provided 
Division/ Discussed Jackets Q U E S T I 0 N Ss Overall 
Program How Review! Reviewed and Evaluation 
Conducted | How Selected (i) (ii) (iii) (iv) (vil lof Stewardship 
T T | 
a bce 
DMR 
~ Polymers x x xX x x O K xX 0 x 
- Ceramics & x xX x x x xX Xx xX 0 x 
Electronic Materials 
- Metallurgy x s x x xX xX x x 0 x 
- 
SZ | RHY 
£ Atomic, Molecular, . x 1 x KX xX x x 0 x 
& Plasma Physics 
L ee 
1 NSF Circular No. 147 imposes no requirement on NSF management to ask additional quest: ons row 


for the review committee to pose ariy. 


© NSF Circular No. 129 (Rev. 3) required program officers on May 29, 1984 to document their own 
evaluation of the quality of the investigators previous research accomplishments. Thais 15 well ant: 
FY 1984, the year most external oversight committees reviewed. 
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Element 1 Element < Element 3 Element 4 
a _ 4 
Reported # S we E Cc I I Cc Provided 
Division/ Discussed Jackets 0 U E S T I N Ss Overall 
Program How Review! Reviewed and Evaluation 
Conducted How Selected (i) (ii) (111) (iv) (v)l lof Stewardshio 
= ——} 
abcé 
ST1IA 
DRII 
- Minerity Research x x x x xX na x x 0 x 
Initiation 
- Research Improvement x x x x x Xx x x 0 x 
in Minoraty Institu- 
tions 
I 
—————E—E ee ——— = ‘ 
l NSF Circular No. 147 imposes no requirement on NSF management to ask additional questions new 
for the review committee to pose any. 
© NSF Circular No. 12€9 (Rev. 3) required program officers on May 29, 1984 to document their own 
evaluation of the quality of the investigators previous research accomplishments. This 18S well ant 
FY 1984, the year most external oversight committees reviewed. 


Key to Table 1: 


Programs under review appear in the far left column. An "“X" indicates the element is fully 


eovered in the review committee’s report; "“/" means part of the element is in the report; 
and “O"“ means the element is missing from the report. 


For specific question (v) under Element 2 additional questions were not always suggested 
by NSF or by the oversight committee so that the absence of any answer to this question 

does not necessarily mean that the review committee failed to address part of its task. 

Similarly the opportunity for the committee to address cther points under Element S was 

not always used and for this reason Table 1 does not report on Elemert 5S. 


NATIONAL SCIENCE FOUNDATION 


Office of the Director 
Washington. D.C. 20550 


NSF CIRCULAR NO. 147 
(Revision 1) 


ADMINISTRATION AND MANAGEMENT 


Subject: External Peer Oversight 


|. Purpose. This Circular sets forth the guidelines for 
external peer oversight of Foundation programs. 

2. Cancellation. This Circular cancels NSF Circular 147, 
dated January 17, 1979. 

3. Policy. 

a. General Policy. Each program shall be reviewed 
periodically by outside experts knowledgeable in the area 
of the program in order to provide NSF management with 
an advisory appraisal of the technical, as distinct from the 
administrative, stewardship by the NSF. This review is in 
addition to the peer reviews of particular proposals. 

b. Oversight Committee Membership. Foundation advi- 
sory committees or panels (or subgroups), augmented by 
additional experts when needed to ensure adequate review, 
may be used to perform cversight review. If an ad hoc 
group ts used, members should be selected in accordance 
with the criteria for selection of advisory committee mem- 
bers described in NSF Circular 132, “Peer Review and 
Guidelines for the Evaluation of Proposals.” It 1s desirable 
that one or more of the experts be from subareas other 
than the suburea being reviewed to ensure that the review is 
conducted in the context of related Foundation programs 
(¢.g., an experimental physicist should be part of the group 
reviewing the Theoretical Physics program). Members of 
the review groups &) Ve appointed in the same manner as 
members of advisory committees. Membership of these com- 
mittees will be published in the NSF Annual Report 

c. Review Cycle. Each program shall be reviewed at 
least once every 3 years. At the beginning of each fiscal 
year, Assistant Directors will prepare a plan for the reviews 
to be conducted that year and submit this plan to the Director 
after discussion with the Office of Audit and Oversight. 
The plan will include a list of all of the programs of the 
Directorate, month and year when each was last reviewed, 
the expected schedules for the reviews to be conducted 
during the year, identification of the reviewing group, and 
the expected year of the nexi review for those programs not 
being reviewed during the current year. 

Where the NSF program structure is not an appropri- 
ate subdivision for a review of this type, the plan submitted 
by an Assistant Director will describe the scope of each 
oversight eview (¢.g., Arctic and Antarctic Geology}. 

d. Access to Files. Committees assembled for program 
oversight will have access to all program documentation, 


S| 
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including preposal jackets on which action has been 
completed, and will review a sample of the awards, declina- 
tions, and withdrawals at the beginning of each meeting. 
Committee members will be reminded that the names and 
comments of ad hoc reviewers must remain confidential. 
No committee members shall review any jacket from the 
sume institution employing the committee member. 

e. Specific Questions. Specific questions will be devel- 
oped and sent to the committee before each review to pro- 
vide a framework for the review. The substance of the 
following questions will be included in all cases: 

(1) Is the review process functioning properly in terms 
of the selection of an appropriate number of technically 
qualified reviewers and adequate quality and freedom from 
bias of the reviewers? (Not applicable to national centers, 
logistics support, etc.) 

{2}(a} Are the decisions of the Program Officer under- 
standable from the documentation accompanying a recom- 
mendation’ 

{2}(b) Are the decisions appropriate as to award or 
declination and as to scope, duration, and size of the project” _ 

(2}(c) Did Program Officers take into account the 
PI's performance on preceding awards? (Documentation 
should be in the grant folder, specifically on NSF Form 7, 
“Review Record.”’} 

(3) Are there any significant imbalances seen in the 
program under review such as size versus number of awards, 
subject matter distribution of awards, age, or geographic 
distribution of Principal Investigators / Project Directors” 

(4) Wuthin the limits of its budget, 1s the program 
meeting NSF objectives? (If possible, oversight review should 
comment on the overall impact of the program and the 
important results coming from the projects supported by 
the program.) 

(5} Are there any additional questions that should 
be addressed in this oversight review? (Assistant Directors 
may wish to ask additional questions such as ones relating 
to priority setting, the balance between the program under 
roview and other programs in NSF or other agencies and to 
changes taking place in a field which will affect future pro- 
gram balance. } 

f. Reports. The oversight committee will report in writ- 
ing to the Director and to the Assistant Director responsi- 
ble for the program. A copy of the report will be sent to the 
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Office of Audit and Oversight. The report is to include 

{1} a brief discussion of the way in which the review 
was conducted: 

{2} the number of jackets reviewed and the way in 
which they were selected: 

(3) the committee's relies to the specific questions 
asked in Se, 

(4) an overall evaluation of the technical steward- 
ship of the Foundation staff in managing the program: and 

(5) any other points the oversight committee wishes 
to address. 

Prior to formal submission of a report, a draft shall be 
examined by the cognizant NSF staff in order to arrange 
for correction of any factual errors. The staff shall prepare 
comments on the report and its conclusions. If desired by 
an Assistant Director, these comments may be attached to 
the report. 

These reports will be submitted within 90 days of the 
meeting of the oversi zht committee, or if the review is made by 
other than an advisory committee and ts to be commented 
on by the appropriate advisory commitice, within 60 days 
of the meeting of the advisory committee. 

g. Availability of Reporis. The reports will be provided 
to the program staff and the appropriate advisory commit- 
tee and will be made available to the scientific community 
and to the public upon request. Copies are filed with the 
Library of Congress for public inspection and use. Assis- 
tant Directors may wish to arrange additional publication 
or dissemination. 

h. Annual Summary. Each year a summary report on 
the oversight reviews will be prepared by the Office of 
Audit and Oversight for the Director to transmit to the 
National Science Board as part of the Director's annual 
report on Foundation use of peer review 


Distribution: E 
Originating Unit: Office of Audit and Oversight 


4. Responsibilities. 
a. Each Assistant Director is responsible for 

{1} planning and conducting oversight reviews of 
the programs in his or her Directorate: 

(2) transmitting the Directorate annual oversight 
review plan to the Director with a copy to the Office of 
Audit and Oversight: 

(4) ensuring that the requirements of the Federal 
Advisory Committee Act, Public Law 92-463 (S U.S.C. 
App. |}. are followed (see NSF Circular 109, “Advisory 
Committee Management}: 

(4) transmitting oversight committee reports to the 
Director with a copy to the Office of Audit and Oversight 
and 10 copies to the Committee Management Officer (DPM) 
for transmittal to the Library of Congress: and 

(5) preparing an appropriate response to the over- 
sight committee under his or her cognizance. 

b. Director, Office of Audit and Oversight, is responsi- 
ble for: 

{1} advising Assistant Direc 
oversight procedures, 

{2} monitoring the external peer oversight process: 

(3) preparing a composite schedule of Directorate 
oversight plans: and 

(4) preparing the annual summary of external peer 
oversight for the Director to transmit to the National Sci- 
ence Board 


Zduveet. 0 basopy- 


Edward A. Knapp 
Director 


_ and their staffs on 


APPENDIX III 


NATIONAL SCIENCE FOUNDATION ANALYSIS 
GEOGRAPHIC DISTRIBUTION OF AGENCY FUNDING 


The following charts are all based on the relative distribution 
of 1983 R & D funds of the National Science Foundation. The 
measure chosen to distribute these funds differs; the first two 
ere commonly used to measure the relative equity of NSF's funding 
patterns against some standard other than science. The fourth, 
and last, is a presentation of the impact of NSF's standard for 
research excellence on its funding distribution. The third, the 
number of scientists and engineers, is an attempt at a halfway 
measure. 


The substance of the analysis displayed by Charts 2, 3, and 4 is 
that when viewed against a number of different, but significant 
parameters, the distribution of NSF funding is clearly more 
equitable than Chart 1 would lead one to conclude. The data on 
Charts 2, 3, and 4 show that almost half of the states exceed or 
meet the average funding level indicated. Further, in many of 
those states where funding levels are significantly below the 
average, NSF is making a cor. sted effort to assist in the 
improvement of the scientific and engineering research and 
education infrastructure through such activities as the 
Experimental Program to Stimulate Competitive Research and 
enhanced support levels for undergraduate institutions in both 
the research and education activities of the Foundation. 
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CHART 1 of 4 


NSF R&B FUNDING TO UNIVERSITIES 


AND COLLEGES BY STATE 


States are arrayed in order of receipts of 1983 NSF P&D dollars 


to universities and colleces. 


California, New York, Massachusetts, and Illinois are the high recipients. 


EPSCOR states represented bv solid bars. 


SOURCE: National Science 
Foundation Feb. 1986 


NSF R&D $ TO UNIV. & COLLEGES CHART 2 of 4 


PER CAPITA BY STATE 
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o Same NSF funding data dividec by each state's vopulation. Plotted in 
in descending order of per cavita state funding. 


o The high states are Massachusetts ($13.56/capvita), Rhode Island, Alaska, 
California, New York, Utah, and Hawaii. 


o States represented by solid bars are the states being assisted through 
EPSCOR. 


o Twenty-two states exceed or meet the average including such states as 
Delaware, Arizona, Wyoming, and New Hampshire. 


SOURCE: National Science 
Foundation Feb. 1986 
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NSF R&D $ TO UNIV. & COLLEGES CHART 3 of 4 
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o NSF R&D funding divided by the number of academic scientists and engineers 
in each state, arranged in descending order. 


o The high states are Alaska, Rhode Island, Massachusetts, and Arizona. 


o States represented by solid bars are the states now being assisted 
through EPSCOR. 


© Twenty states exceed or meet the average including such states as 
Colorado, Minnesota, and Utah. 


SOURCE: National Science 
Foundation Feb. 1986 
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NSF R&D $ TO UNIV. & COLL. BY STATE 


PER INDEX OF HIGH QUALITY PROPOSALS CHART 4 of 4 
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NSF R&D FUNDING TO UNIVERSITIES AND 
COLLEGES DIVIDED BY NUMBER OF HIGH 


RATED PROPOSALS PER STATE 


AVERAGE (PROPOSAL QUALITY INDEX) 
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o Total NSF R&D funding per state (1983) divided by the number of high-rated* proposals from 
each state in a sample of 3200 proposals reviewed in 1980, 1981, and 1982. 


0 The chart shows that, in general, NSF's allocation of R&D allocation of R&D obligations to 
academic institutions is proportional to the highly rated proposals coming from a state. 


o States represented by solid bars are the states being assisted through EPSCOR. 
0 Nineteen states exceed or meet the average including such states as Nebraska, Arka‘sas, 


Mississippi, Montana, Florida, and North Carolina. 


SOURCE: National Science Foundation Feb. 1986 


* "High-rated" = N highest-rated proposals in each Division where N = number of awards 
in that Division. The total is 1222. 


57 


” 


APPENDIX IV 


Short bibliography cf empirical studies of NSF peer review 


Wilson, M.K., "The Top Twenty and the Rest: Big Chemistry and Little 
Funding,” Annual Review of Physical Chemistry 26: 1-16, 1975. 


D. Hensler, “Perceptions of the NSF Peer Review Process: A Report on a Survey 
of NSF Reviews and Applicants," prepared for the Committee on Peer Review of 
the National Science Board, 1976. 


"Reviewer and Proposer Similarity and Its Effect on Award Decision,” NSF 
Evaluation Staff Studies 76-1. 


Cole, S., L. Rubin and J.R. Cole, Peer Review in the National Science 
Foundation, Phase One of a Study (Washington, D.C.: National Academy of 
Sciences, 1978). 


DeWitt, T., R.S. Nicholson, and M.K. Wilson, "Science Citation Index and 
Chemistry," Scientometrics 2(4): 265-275, 1980. 


Cole, J.R., and S. Cole, Peer Review in the National Science Foundation, Phase 
Two of a Study (Washington, D.C.: National Academy of Sciences, 1981). 


"Geographic and Institutional Effect on the Award Decision,” NSF Evaluation 
Staff Studies 81-2. 


"A Study of the Fairness of the NSF Award Decision Process," NSF Evaluation 
Staff Studies 82-1. 


Wall, R., “Peer Review in the Ocean Sciences Research Section of the National 
Science Foundation,” EOS 63(December 7):1202, 1982. 


"Fairness of the NSF Award Decision Process," NSF Evaluation Staff Studies 84- 
Be 


“The NSF Post-performance Evaluation Study," NSF Evaluation Staff Studies 84- 
2. 


"Fairness of the NSF Award Decision Process," NSF Evaluation Staff Studies 85- 
1. 


Porter, A.L. and F.A. Rossini, “Peer Review of Interdisciplinary Research,” 
Science, Technology, and Human Values 10(3), summer 1985. 


APPENDIX V 


COMPOSITION OF NSF ADVISORY COMMITTEES 


(Data as of November 30, 1985) 


January 1986 


Prepared by the 
Division of Personnel 
and Management 


INTRODUCTION 


The Third report on the composition of NSF advisory committees and panels 
contains data as of November 30, 1985. The two previous reports contained 
data as of March 31, 1985 and September 30, 1984. All data were provided 
by committee management contacts of NSF advisory committees and panels, 
whose assistance is greatly appreciated. 


The report is divided into two parts: (1) analyses and summaries and (2) 
aggregated data and input sheets. With the advent of personal computers, 
the data have been manipulated with a greater depth and breadth of analysis 
than in previous reports. A large amount of Part I includes computer- 
produced charts and statistical computations and analyses of these data. 


Questions about this report should be addressed to Gail McHenry, Management 
Analysis Section, Division of Personnel and Management, 357-9520. 


TECHNICAL NOTES 


There are six questions in the report input sheet. Below is the 
description of the source and definitions. 


1. Field of Study. There are 17 major fields and an “other" 
category. The fields are based on the Central Personnel Data 
File (CPDF) and were modified for applicability to NSF. 

2. Geographic Region. The five regions are: 


NORTHEAST = NY. NJ. PA, Gre ME, MA. NH, RI, VT. 


SOUTH - DE. DC. MD. VA, WV, NC, SC, GA, FL. AL. LA, KY. MS, 
TN. TX. AR, OK. 


WEST - AK, CA. HI. OR, WA, ID, AZ. CO. MT. NM, UT. WY, NV. 
MIDWEST - IA. KS. MO. MN, NE, ND, SD. IL. IN. MI. OH, WI. 
FOREIGN - Any country outside the 50 United States. 


3. Age oo; Members. The age category is divided into “under 4O 
years" and “4O years and over." 


a. Sex. Self-explanatory,. programs classify members. 


b. Race. Race of members is reported using the following 
Office of Personnel Management (OPM) categories: 


American Indian or Alaskan Native: A person 
having origins in any of the original peoples of North America, 
and who maintains cultural identification through community 
recognition or tribal affiliation. 


Asian or Pacific Islander. A person having 
origins in any of the original peoples of the Far East Southeast 
Asia, the Indian subcontinent, or the Pacific Islands This area 
includes. for example. china, Japan, India, Sores, th. hilippine 


Islands, and Samoa. 


Black. not of Hispanic Origin. A person hnaving 
origins in any of the black racial groups of Africa. Does not 
include persons of Mexican, Puerto Rican, Cuban, Central or South 
American, or other Spanish cultures or origins (see Hispanic). 


Hispanic. A person of Mexican, Puerto Rican, 
Cuban. Central or South American, or other Spanish cultures or 
origins. Does not include persons of Portuguese culture or origin. 
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White. not of Hispanic Origin. A person hnaving 
crigins in any or the original peoples of Europe. North Africa, 
or the Middle East. Does not include persons of Mexican. Puerto 
Rican. Cuban. Central or South American, or other Spanish 
cultures or origins isee Hispanic). Also includes persons not 
included in otner categories. 


Cc. Handicapped. Based on the OPM d2finitions. “handi- 
cappec™ includes persons with: speech. hearing. and vision 
impairments: missing extremities: nonparalytic orthopedic 
impairments: and partial or complete paralysis. 


5. Affiliation. Tne categories in affiliation are based on the 
NSF code compendium, “Institutional/Technical Reference File 
(ITRF)."” and refined with “predominantly undergraduate 
institutions (PUI‘s)” and more distinct non-academic institutions 
such as “research institutes” and “National centers." The 
Gefinition of PUI‘s is based on information provided by the 
Division of Research Initiation and Improvement (RII). STIA. DPM 
reviewed initial ciassification provided by programs in 
accordance with the ITRF and with RII and adjusted classifi- 
cations where necessary. 


6. Ter 10 Institutions. This consists of 10 academic insti- 
tutions which ranked in the top 10 for annualized dollar amount 


of NSF awards. The FY 1984 and FY 1985 top 10 are the same 
@lthough the relative ranking changed. The rankings for FY ‘&84& 
anda ‘85 are: 


FY 1984 
MIT 
University of California at San Diego 
Cornell 
University of Illinois at Urbana 
columbia 
Stanford 
University of Washington 
University of California at Berkeley 
University of Wisconsin at Madison 
Cal Tech 
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FY 1985 
MIT 


Cornell 

University of Illinois at Urbana 
University of California at San Diego 
Univereity of California at Berkeley 
Columbia 

Stanford 

University of Washington 

Cal Tech 

University of Wisconsin at Madison 
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COMPOSITION OF NSF ADVISORY COMMITTEES 
ANALYSIS 


i. Total Number of Members. The November 30, 1985 reporting 
period shows an 11.7% (77 members) increase ir total membership 
over the March reporting period. This is due primarily to the 
appointment of additional members to the Engineering Directorate 
committees. 


2. Typical Member. There is little change in the “typical 
member™ profile. Below are the data is of November 30, i985: 


White (92%) 

Male (82%) 

40 Years Old or Older (79%) 

From an Academic Institution (73%) 

From the Northeast (34%) or West (27%). and 
Not Handicapped (99%). 


3. Race of Members. The proportion of Blacks and Asians/Pacific 
Islanders has grown eince March ‘85 and the percentage of Blacks 
has increased in each reporting period (2.7%. 2.9%, 3.3%). 
Hispanic representation. on the other hand, has decrease from 13 
to 8 members. In March, we reported three American Indian/ 
Alaskan members which increased to four in November. 


See chart i (page 6) for statistics comparing non-white repre- 
sentation during the three reporting periods and chart 2 (page 7) 
for a’ display of the four committees reporting the largest 
proportion of non-white members. 


a. Handicapped Members. There are five handicapped members, a 
Gecline of one since the last reporting period. 


5. Affiliation. The primary affiliation of NSF advisory 
committee/panel members is the academic institution. However. 
the proportion has decreased slightly during this reporting 
period (from 76% in March ‘85 to 74%). 


a. Academic Representation. Over the three reporting 
periods, representation from the top 10 schools (in terms of NSF 
awards) has steadily increased while the percentage of members 
from other academic institutions and predominantly undergraduate 
institutions (PUI‘s) has declined. (See chart 4 on page 9 for a 
comparison of distribution of academic membership over the three 
reporting periods. ) 
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(1) Top 10. The first pie chart on chart 5 (page 10) 
shows the contribution of each directorate to the representation 
cof top 10 schools. The percentages from the directorates closely 
resembled their proportion of members to the total membership 
except STIA which contributed less proportionately and MPS which 
contributed more. 


Of all the directorates, MPS had the largest percentage from top 
10 schools (30%) and BBS reported the most members (47) from the 
top 10. Over the three reporting periods. there has been a 
steady decline in representation from the University of 
California at Berkeley while representation from the other 
schools nas fluctuated. 


(2) Predominantly Undergraduate Institutions (PUI's). 
Over the three reporting periods, membership from PUI'‘'s has 
decreased slightly. (See chart 4 on page 9.) The table on Chart 
5 (page 10) shows the organizational representation from PUI's 
(12 members). 


(3) Other Academic Affiliation. This category 
consists of members from academe who are not included in the top 
10 schools or PUI's. BBS represents the largest proportion of 
members from “other academic" organizations (about 73% of BBS 
membership). AAEO reported about half of its membership in this 
category (49%). The remaining directorates reported between 25% 
to 44x of their membership as “other academic." 


b. Nonacademic Representation. Almost 27% of the member- 
snip is affiliated with an organization other than an academic 
institution. See chart 6 (page 11) for depiction of the nonaca- 
demic membership over the three reporting periods. Below is a 
Gescription of members from the largest categories. 


(1) Large Corporations. Except for the representation 
ef government in March 1985, large corporations have dominated 
the nonacademic membership. In fact, this category represents 
almost double the percentage over the last reporting period (from 
6% to almost 11%). The most significant representation during 
this reporting pericd is from Bell Labs (16% of the large 
corporation members) and du Pont, Exxon, and IBM (5% each of the 
large corporation membdership). The remaining number of members 
affiliated with large corporations (54 members) represent 42 
other organizations. 


Almost nalf of the representation from large corporations (UUX oF 
total representation) serves on the Engineering committees. The 
next two highest percentages from large corporations (14% each) 
are from committees reporting to MPS and the Offfce of the 
Director. STIA contributes 12% of the total members from large 
corporations, representing 22% of STIA's members. 


(2) Government. This category represents a consolid- 
ation of several governme ital organizations: Federal government, 
State and local government, and government laboratories. Govern- 
ment affiliation represents 22% of the nonacademic membership. 
The total government representation dropped steadily from around 
8x to 6%. The only component of the government figure that did 
not decrease from ‘84 to this reporting period was State and 
local government which doubled its representation. 


Approximately 25% of the government representation is from BBS 
committees and panels, accounting for 4X of BBS total member- 
ship. The Advisory Committee for SEE and committees reporting to 
the Office of the Director represent the greatest proportion of 
their membership from all components of government (13% each). 


(3) Research Institutes. Representation from the 
research institutes has fluctuated over the reporting periods. 
This reporting period shows a decline of almost 2% of the 
membership. The largest representation (three members) is from 
Words Hole Oceanographic Institution. The remaining 27 members 
are from 22 different research institutes. 


Almost half of the members from research institutes serve on the 
BBS committee and panels, amounting to 4.5% of BBS’ membership. 
The largest percentage of research institutes' representation in 
a directorate is in ADM where 21% (or 6 members) of ADM members 
are from research institutes. 


6. Geographic Distribution. Chart 7 (page 12) shows the slight 
changes in representation from the four geographic regions. The 
South and foreign countries nave steadily declined while the 
other regions have fluctuated. Northeast still represents the 
largest percentage followed by West, together comprising over 
half of the membership. 
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SLIMMAFRY OF NSF ADVISORY COMMITTEES 
Number of Advisory Committees and Panels 


Tyre of Committee/Panel 
Sponsoring Total Program Ppl/Appl Cognizant 


Urgenization (% of TOT) Advice Review Mffice 
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“iffice of the 


Director: a 3 Office of the Director 

6+ 2% i Ofc/Advcd Sci- Computing 
ADM 2 v4 ADM <cDPM) 

Ze 1% 
AAEU 7 S Division level 

10.8% 1 QCE/USR Section 

1 EAR Sections 

BES a4 1 AD/BBS 

S22 3% 33 Section/Program level 
ENG = 1 AD/ENG 

12.3% d Division level 

Z Section level 
1 Ofc/Interdiscp. Res- 

MPS 5 5 Division level 

7-7% 
SEE 1 i AD/SEE 

1.5% 
STIA d 3 Division level 

be 2% i EVIST Program 
TOTAL NSF 65 27 32 
Percentage 100.0% 41.5% 58.5% 
Totals from 65 26 39 
March ‘Ss 100.0% d0O.0% 60.0% 
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(No. Meabers) 


§ Meabers 


TOTAL NSF STATISTICS 


NSF ADVISOFY COMMITTEE MEMBEFSHIP 


NATIONAL SCIENCE FOUNDATION 


Study Catecory 


Aq. & Natural Pesources 
Biological Science 
Business 4 Manageeent 
Computer 4 Info. Science 
Education 

Engineering 

Health Proressions 

Law 

Matheaatics 

Physics 

Cheaistry 

Astronoay 

Atmospheric Sci 

Earth Science 

Ocean Science 
Psychology 

Social Sciences 

Other 


1 @ Meabers 


1.6% 
71 
4.1% 
10-64% 
1.8% 
2.3% 
1-13 
1.8% 


100.0% 


Affiliation 


Res. Inst. 

Larae Coro- 

Saal! Business 
17 = Fed. Govt. 

§ State/Local Govt 
Govt. Lab 

2 National Center 
Nonorofit Inst. 
tts Individual 

8 Other 


t 


100.0% 


SEX 
Male 
Ferale 


# Meabers 
| 


Predom. Undergrad. Inst. 
Other Acadeaic Inst. 
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# Meabers Top 10 Acadesic 
13) OMIT 
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APPENDIX VI 
NSF Reconsiderations -- CY1983 


Orifice of Audit and Oversignt 


Msy 3, 1984 


Director, OAO 


NSF Reconsiderations - CY 1983 


Addressees 


Attached for your information is an OAO oversight study for the 
NSB Committee on Audit and Oversight on NSF Reconsiderations -- 
CY 1983. Ho major problems surfaced although there were minor 
problems involving (1) timeliness of NSF responses and (2) the 


orderliness of reconsideration documentation. 


Zs/ Jerome H. Fregeau 


Jerome H. Fregesu 
Attachment 
Addrescsees: 


Dr. Knapp, Director 

Dr. Nicholson, Acting Deputy Director 

Dr. Bridgewater, Acting Assistant Director/AAEO 
Dr. Rabin, Acting Assistant Director/BBS 

Dr. Hall, Acting Assistant Director/E 

Dr. Bardon, Acting Assistant Director/MPS 

Dr. Bautz, “ting Assistant Director/SEE 

Mr. Green, Avsistent Director/STIA 

Mr. Ubois, sistent Director/A 


bec: Dr. Zwolenik 
Mrs. Thomas 
CHRON 
FILE: NSB/AO-84-15, RECONSIDEATIONS--CY1983 (NSB/AO 29th 
Meeting—05/10/84). 


OAO: JJZ: DFH:05/03/84:357-9458... TAPE #13, #6... 
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NSB/AO-84-15 


April 30, 1984 


RECONSIDERATIONS CY 1983 


Introduction 


Although the NSF permits no formal rebuttal process during the review of 
a proposal,— it has, since January 1976, provided for reconsideration 
of a declined proposal at three management levels. A principal investi- 
gator (PI) whose proposal has been declined may (1) obtain an 
explanation of the declination from the responsible Program Officer. If 
this explanation does not satisfy the PI that the proposal was fairly 
handled and reasonably evaluated, the PI may also (2) obtain recon- 
sideration of the declination by the responsible Assistant Director. If 
the applicant institution is still not satisfied after reconsideration 
by the responsible Assistant Director, it may (3) obtain further 
reconsideration by the Deputy Director of the NSF. If a proposal has 
been declined following review by the National Science Board, however, 
only an explanation is available. Details on the reconsideration 
process are in the attached NSF Circular 127. 


Although any explanation given to the PI by a Program Officer is the 
subject of a diary note in the proposal jacket, ther are no NSF-wide 
data reported on this level of reconsideration.— However, each 
directorate must record all requests for reconsideration by the 
Assistant Director as specified in section 5 of nSF Circular 127 and 
forward these records quarterly to the NSF Deputy Director. The Office 
of the Deputy Director maintains similar records on requests it receives 
for further reconsideration from applicant institutions, 


As part of its oversight activities, OAO reviewed the quarterly reports 
of each directorate: and of the Office of the Deputy Director for CY 
1983. It also reviewed the document record in a sample of the proposal 
jackets for which there were requests to the Assistant Directors for 
reconsideration. No major problems were found. However, timeliness in 
cvurmishing the results of the reconsideration process and the orderly 
maintainence of reconsideration records in jackets are minor problems 
worthy of corrective attention. 


1/ However, the Program Officer can ask for additional information, 
response to reviewer comments, etc. if needed to make a decision. 


2/ Anecdotal information indicates that the most popular method for 
reversing a declination is submission of a new proposal which takes 
account of information and criticism obtained from discussion with 
the Program Officer and from anonymous verbatim peer reviews. 
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Statistical Data 


In CY 1983, Assistant Directors reconsidered 44 declinations 
distributed: 


AAEO 
BBS 
ENG 
MPS 
SEE 
STIA 


fFonwn uu & 
> 


These 44 are 0.4% of the 10,168 proposals NSF declined in CY 1983. 


In all completed reconsiderations except one, the declination was 
upheld. The Acting Assistant Director/BBS reversed one declination. 


Four of these 44 were further reconsidered by the Office of the Deputy 
Director who upheld the original declination and the Assistant 
Director's decision to sustain that declination. 


Furnishing Results of Reconsideration 


NSF Circular 127 requires that those requesting reconsideration from 
the Assistant Director or from the Deputy Director receive the results 
of the reconsideration within 30 days. 


It was not possible to determine the timeliness of STIA's response to 
the four requests for reconsideration by the Assistant Director since 
STIA did not provide the “date of receipt" of each request in its 
quarterly reports. Future STIA reports should provide the information 
listed in section 5, NSF Circular 127. 


Of the 40 requests to Assistant Directors other than STIA, 50% waited, 
or were waiting, for the results of the Assistant Director's recon- 
sideration with no interim notification of delay for more than 30 days. 
Although there was one instance each of 70- and 85-day delays without 
any interim notification ost overdue replies fell in the 30- to 45-day 
category. Whenever the results of the reconsideration cannot be 
furnished within 30 days, the cognizant Assistant Director should 
furnish the PI a written explanation of the need for more time and 
indicate the date by which the results of the reconsideration can be 
expected--as required in section 4 of NSF Circular 127. 


Of the 4 institutional requests for reconsideration by the Deputy 
Director, 2 received results within 30 days and 2 withim 36 days. 
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Sample of Jackets Under Reconsideration 


OAO reviewed 12 proposal jackets from among the 44 requesting recon- 
sideration by the Assistant Directors. The distribution was: 


AAEO - 2 
BBS - 1 
ENG - 2 
MPS - 6 
SEE - 0 
STIA - l 


Of these 12, 11 declinations were upheld at the Assistant Director's 
level and 1 declination was reversed by the Acting Assistant 
Director/BBS. Two MPS declinations that were upheld at the Assistant 
Director's level appealed to the Deputy Director who sustained the 
declinations. Both of these were submissions from small companies. 


In general, there seems to be confusion in the jacket with respect to 
reconsideration documents. These are confidential review documents 
and should be filed chronologically on the left side of the jacket. 


In 4 instances the Assistant Director reconsidered without the PI 
having sought 4nd obtained an explanation from the Program Officer as 
is normally expected. 


Based on the original ratings and reviews found in each of the 12 
jackets, it is difficult to understand why approximately 40% 
requested reconsideration at the Assistant Director level given the 
obviously low ratings of the reviewers and the current high cutoff 
levels. It is equally difficult to understand why an even higher 
percentage requested reconsideration from the Office of the Deputy 
Director. Leaving aside these cases which are well below the cutoff 
region, the others appear to have some plausible basis upon which the 
PI might, if correct, feel that a reversed decision was warranted. 


At the Assistant Director level and fully in accord with the require- 
ments of NSF Circular 127, the AD performed the requested 
reconsideration and/or appointed one or two others not involved with 
the original evaluation of the declination to perform. the 
reconsideration. The same was true for reconsiderations by the Deputy 
Director. 


The records support quite well the decisions made by the ADs and the 
Deputy Director. 


In general the correspondence furnishing the results of the recon- 
sideration process shows a commendable sensitivity to the situation. 
In parts of three replies, bold restatements of the basis for declina- 


tion appear somewhat harsh and/or patronizing, but these are not 
typical. 
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Conclusion 


Aside from the problems of timeliness and document filing, the 
reconsideration process introduced into NSF in January 1976 appears 
to be operating smoothly and effectively, thereby contributing 
positively to PI and public confidence in the NSF peer review system. 


Jem} Brrtimd 


James J. Zwolenik 
Senior Staff Associate-Oversight 
Office of Audit and Oversight 


Attachment: NSF Circular 127 
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NATIONAL SCIENCE FOUNDATION 


Office of the Director 
Washington, D.C. 20550 


NSF CIRCULAR NO. 127 
(Revision No. 2) 


GRANTS ANDCONTRACTS 


August 8, 1980 


Subject: Reconsideratioa of Proposals Declined or Returned by NSF 


|. Purpose. This Circular implements the provisions 
of Important Notice No. 84, “Reconsideration of Pro- 
posals Declined or Returned by the NSF,” dated 
August 8, 1980, which establishes procedures and 
responsibilities for the reconsideration of proposals 
for grant assistance declined or returned by NSF. This 
Circular does not apply to procurements governed by 
the Federal Property and Administrative Services Act 
or to applications for fellowships or travel grants. 

2. Cancellation. This Circular cancels NSF Circular 
No. 127, Revision No. |, dated October 2, 1979. 

3. Policy. Award of NSF assistance is discretionary. 
Nonetheless, 4 principal investigator (P1) whose pro- 
posal has been declined may obtain an explanation of 
the declination from the responsible Program Officer. 
If this explanation does not satisfy the PI that the 
proposal was fairly handled and reasonably evaluated. 
the PI may also obtain reconsideration of the declination 
by the responsible Assistant Director. If the applicant 
institution is still not satistied after reconsideration 
by the responsible Assistant Director, it may obtain 
further reconsideration by the Deputy Director of the 
NSF. If a proposal has been declined following review 
by the National Science Board, however, only an ex- 
planation will be available. A Pl whose proposal has 
been returned because it is inappropriate for consider- 
ution by NSF may also request reconsideration of this 
determination. 

Reconsideration is not an adversary process and 
no formal hearing is provided. The Foundation cannot 
ussure upplicants that reconsideration will result in 
un award even if error is established in connection 
with the initial evaluation. 

4. Procedures. 

a. Explanation by Program Officer. A Pi who 
wishes information about a declination or returned 
proposal may request an explanation from the appro- 
priate Program Officer. That may be done by letter, by 
telephone, of in person. The Program Officer will ex- 
plain the basis for the declination or the return. On 
request the Program Officer will provide the PI ver- 


batim copies of peer reviews received on the declina- 
tion with only the names and other identifying data of 
the individual reviewers deleted. The Program Officer 
will afford the Pl an opportunity to present the PI's 
point of view and will take any further action that 
seems appropriate. 

No revisions made to a proposal after declination 
will be considered in connection with the original 
proposal. However, a substantially revised proposal 
may be submitied for review and evaluation as a new 
proposal under the usual procedures. 

b. Reconsideration by the Assistant Director at the 
Request of the Principal Investigator. \f dissatisfied 
with the Program Officer's explanation, the PI may 
request in writing that the Foundation reconsider its 
action. Such a request will be considered only if the PI 
has first sought and obtained an explanation from the 
Program Officer and only if the request is received 
by the Foundation within 180 days after the declina- 
tion or return. 

(1) The PI should address the request to the 
Assistant Director for the Directorate that handled 
the proposal and should explain why the PI believes 
that the declination or return was unwarranted. 

(2) The Assistant Director will consider the 
request for reconsideration. The record of the evaluation 
of the declined proposal will be reviewed to determine 
whether the declined proposal was fairly handled and 
reasonably evaluated. The returned proposal will be 
reviewed to determine whether the proposed research 
is inuppropriate for NSF consideration. 

(u) The Assistant Director may request addi- 
tional information from the PI and may obtain addi- 
tional peer reviews. 

(b) The Assistant Director may conduct the 
reconsideration personally or may designate another 
NSF official who had no part in the initial evaluation 
to do so. As used here, “Assistant Director” includes 
such a designated official. 

(3) Within 30 days after the date of the request. 
the Assistant Director will furnish the Pl in writing the 
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results of the reconsideration. If results cannot be 
furnished within 30 days, the Assistant Director will 
furnish the PI a written explanation of the need for 
more time, indicating the date when results can be 
expected. If the Assistant Director reaffirms the 
declination or return, he or she will inform the Pl 
that the PI's institution may obtain further recon- 
sideration by the Deputy Director. 

c. Further Reconsideration by the Deputy Director 
at the Request of the Applicant Institution. Within '!80 
days after a reconsideration at the Assistant Director 
level, the applicant institution may obtain further re- 
consideration by the Deputy Director of the NSF. 
Only the applicant institution may request further 
reconsideration by the Deputy Director. except that a 
PI who submitted a proposal as an individual, without 
connection with any institution, may submit a request 
for further reconsideration as an individual. 

(1) The institution should address a request for 
further reconsideration to the Deputy Director of the 
NSF. The request need not be in any particular format. 
but it must be in writing and must be signed by the 
institution's president or other chief executive officer 
and by the PI. For declinations, it should explain why 
the institution believes that an error may have occur- 
red in the initial evaluation and why it is not entirely 
satisfied with the reconsideration by the responsible 
Assistant Director. For returned proposals, it should 
explain why the institution believes that an error may 
have occurred in the initial determination that the 
proposal was inappropriate for consideration by NSF. 

(2) The Deputy Director will review the request 
for further reconsideration and the record of earlier 
NSF actions, including the original evaluation of the 
declined proposal and the reconsideration by the As- 
sistant Director. to determine whether the declined 
proposal was fairly handled and reasonably evaluated 
or whether the returned proposal is outside the domain 
of NSF. 

(a) The Deputy Director may request addi- 
tional information from the PI or the applicant in- 
Stitution and may obtain additional peer reviews. 
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(b) The Deputy Director may conduct the 
further reconsideration personally or may designate 
another NSF official who had no part in the initial 
evaluation of the proposal or the earlier reconsidera- 
tion to do so. As used here. “Deputy Director™ in- 
cludes such a designated official. 

(3) Within 30 days after a request for further 
reconsideration is received ai the NSF, the Deputy 
Director will furnish the institution in writing the 
results of the further reconsideration. If results can- 
not be furnished within 30 days, the Deputy Director 
will furnish the institution a written explanation of 
the need for more time. indicating the date when the 
results cun be expected. 

(4) After reconsideration by the Deputy Di- 
rector, u declination or return will not be reconsidered 
further. However, a substantially revised proposal 
may be submitted for review and evaluation as a new 
proposal under the usual procedures. 

5. Reporting Requirement. Each directorate will 
muintain a record of all requests for reconsideration 
by the Assistant Director. The record should include 
the date of receipt. the name of the Pl, the name of the 
applicant institution, and the proposal number and 
title. When the reconsideration is completed, the record 
should be updated to indicate when the results of the 
reconsideration were furnished to the PI and what the 
results were. Within 5 working days after the end of 
each quurter of the fiscal year, each directorate will 
forward to the Deputy Director a copy of the record 
for all requests received and reconsiderations com- 
pleted during the previous quarter. 

The Office of the Deputy Director will maintain 
similar records on requests for further reconsidera- 
tion from applicant institutions. 


Donald N. Lungenberg 
Acting Director 
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